1 | Sparse Directories Support in Subversion |
---|
2 | (a.k.a. "sparse checkouts" / "incomplete directories") |
---|
3 | |
---|
4 | Contents |
---|
5 | ======== |
---|
6 | |
---|
7 | 0. Goals |
---|
8 | 1. Design |
---|
9 | 2. User Interface |
---|
10 | 3. Examples |
---|
11 | 4. Implementation Strategy |
---|
12 | 5. Compatibility Matters |
---|
13 | 6. API Changes |
---|
14 | 7. Work Remaining |
---|
15 | |
---|
16 | 0. Goals |
---|
17 | ======== |
---|
18 | |
---|
19 | Many users have very large trees of which they only want to |
---|
20 | checkout certain parts. In Subversion <= 1.4, 'checkout -N' is not |
---|
21 | up to this task. |
---|
22 | |
---|
23 | Subversion 1.5 introduces the idea of "depth" (controlled via the |
---|
24 | '--depth' and '--set-depth' options) as a replacement for mere |
---|
25 | non-recursiveness (formerly controlled via the '-N' option). Depth |
---|
26 | allows working copies to have exactly the contents the user wants, |
---|
27 | leaving out everything else. |
---|
28 | |
---|
29 | 1. Design |
---|
30 | ========= |
---|
31 | |
---|
32 | We have a new "depth" field in .svn/entries, which has (currently) |
---|
33 | four possible values: depth-empty, depth-files, depth-immediates, |
---|
34 | and depth-infinity. Only this_dir entries may have depths other |
---|
35 | than depth-infinity. |
---|
36 | |
---|
37 | depth-empty ------> Updates will not pull in any files or |
---|
38 | subdirectories not already present. |
---|
39 | |
---|
40 | depth-files ------> Updates will pull in any files not already |
---|
41 | present, but not subdirectories. |
---|
42 | |
---|
43 | depth-immediates -> Updates will pull in any files or |
---|
44 | subdirectories not already present; those |
---|
45 | subdirectories' this_dir entries will |
---|
46 | have depth-empty. |
---|
47 | |
---|
48 | depth-infinity ---> Updates will pull in any files or |
---|
49 | subdirectories not already present; those |
---|
50 | subdirectories' this_dir entries will |
---|
51 | have depth-infinity. Equivalent to |
---|
52 | today's default update behavior. |
---|
53 | |
---|
54 | The new '--depth' option limits how far an operation descends, and |
---|
55 | the new '--set-depth' option changes the depth of a working copy tree. |
---|
56 | |
---|
57 | The new options are explained in more detail in 'Usage' below, but |
---|
58 | these two concepts will aid understanding: |
---|
59 | |
---|
60 | "ambient depth" -----> The depth, or combination of depths, |
---|
61 | of a given working copy. |
---|
62 | |
---|
63 | "requested depth" ---> The depth the user requested for a |
---|
64 | particular operation (e.g., checkout, |
---|
65 | update, switch). This is sometimes |
---|
66 | called the "operational depth". |
---|
67 | |
---|
68 | When when running an operation in a working copy, the requested |
---|
69 | depth never goes deeper than the ambient depth. For example, if |
---|
70 | you run 'svn status --depth=infinity' in a working copy directory |
---|
71 | that was checked out with '--depth=immediates', the status will go |
---|
72 | as far as depth immediates. That is, it descends all the way |
---|
73 | ("infinity") into what's available, but in this case what's |
---|
74 | available is shallower than infinity. |
---|
75 | |
---|
76 | 2. User interface |
---|
77 | ================= |
---|
78 | |
---|
79 | Quick start: |
---|
80 | |
---|
81 | Run checkout with --depth=empty or --depth=files. When you need |
---|
82 | additional files or directories, pull them in with 'svn up NAME' |
---|
83 | (passing --depth for directories as appropriate). |
---|
84 | |
---|
85 | Not-so-quick start: |
---|
86 | |
---|
87 | checkout without --depth or -N behaves the same as it does today, |
---|
88 | which is the same as with --depth=infinity. |
---|
89 | |
---|
90 | checkout --depth=(empty|files|immediates) creates a working copy |
---|
91 | that is (empty | has only files | has files and empty subdirs). |
---|
92 | |
---|
93 | Inside such a working copy, running 'svn up' by itself will update |
---|
94 | only what is already present, but running 'svn up OMITTED_SUBDIR' |
---|
95 | will cause OMITTED_SUBDIR to be brought in at depth-infinity, while |
---|
96 | the rest of the parent working copy remains at its previous depth. |
---|
97 | |
---|
98 | The --depth option limits how far an operation recurses. The |
---|
99 | operation will reach whatever is inside the intersection of the |
---|
100 | ambient depth and the requested depth (see 'Design' section for |
---|
101 | definitions). |
---|
102 | |
---|
103 | The --depth option never changes the depth of an existing dir. |
---|
104 | Instead, use the new '--set-depth=NEW_DEPTH' option for that. |
---|
105 | Right now, --set-depth can only extend (that is, make deeper) a |
---|
106 | directory. (In the future, it will also be able to contract; see |
---|
107 | issue #2843.) We disallow '--depth' and '--set-depth' together. |
---|
108 | |
---|
109 | The -N option has been deprecated, but still works: it simply maps |
---|
110 | to one of --depth=files, --depth=empty, or --depth=immediates, |
---|
111 | depending on context, for compatibility. For most commands, it's |
---|
112 | --depth=files, but for status it's --depth=immediates, and for |
---|
113 | revert and add it's --depth=empty, to be compatible with the |
---|
114 | varying behaviors -N had across these commands. |
---|
115 | |
---|
116 | 'svn info' lists depth, iff invoked on a directory whose depth is |
---|
117 | not the default (depth infinity). |
---|
118 | |
---|
119 | 3. Examples |
---|
120 | =========== |
---|
121 | |
---|
122 | svn co http://.../A |
---|
123 | |
---|
124 | Same as today; everything has depth-infinity. |
---|
125 | |
---|
126 | svn co -N http://.../A |
---|
127 | |
---|
128 | Today, this creates wc containing only mu. Now, this will be |
---|
129 | identical to 'svn co --depth=files /A'. |
---|
130 | |
---|
131 | svn co --depth=empty http://.../A Awc |
---|
132 | |
---|
133 | Creates wc Awc, but empty: no files, no subdirectories. |
---|
134 | |
---|
135 | Awc/.svn/entries this_dir depth-empty |
---|
136 | |
---|
137 | svn co --depth=files http://.../A Awc1 |
---|
138 | |
---|
139 | Creates wc Awc1 with all files (i.e., Awc1/mu) but no |
---|
140 | subdirectories. |
---|
141 | |
---|
142 | Awc1/.svn/entries this_dir depth-files |
---|
143 | ... |
---|
144 | |
---|
145 | svn co --depth=immediates http://.../A Awc2 |
---|
146 | |
---|
147 | Creates wc Awc2 with all files and all subdirectories, but |
---|
148 | subdirectories are empty. |
---|
149 | |
---|
150 | Awc2/.svn/entries this_dir depth-immediates |
---|
151 | B |
---|
152 | C |
---|
153 | Awc2/B/.svn/entries this_dir depth-empty |
---|
154 | Awc2/C/.svn/entries this_dir depth-empty |
---|
155 | ... |
---|
156 | |
---|
157 | svn up Awc/B: |
---|
158 | |
---|
159 | Since B is not yet checked out, add it at depth infinity. |
---|
160 | |
---|
161 | Awc/.svn/entries this_dir depth-empty |
---|
162 | B |
---|
163 | Awc/B/.svn/entries this_dir depth-infinity |
---|
164 | ... |
---|
165 | Awc/B/E/.svn/entries this_dir depth-infinity |
---|
166 | ... |
---|
167 | ... |
---|
168 | |
---|
169 | svn up Awc |
---|
170 | |
---|
171 | Since A is already checked out, don't change its depth, just |
---|
172 | update it. B and everything under it is at depth-infinity, |
---|
173 | so it will be updated just as today. |
---|
174 | |
---|
175 | svn up --depth=immediates Awc/D |
---|
176 | |
---|
177 | Since D is not yet checked out, add it at depth-immediates. |
---|
178 | |
---|
179 | Awc/.svn/entries this_dir depth-empty |
---|
180 | B |
---|
181 | D |
---|
182 | Awc/D/.svn/entries this_dir depth-immediates |
---|
183 | ... |
---|
184 | Awc/D/G/.svn/entries this_dir depth-empty |
---|
185 | ... |
---|
186 | |
---|
187 | svn up --depth=infinity Awc |
---|
188 | |
---|
189 | Update Awc at depth-empty, and Awc/B at depth-infinity, since |
---|
190 | those are the ambient depths of those two directories already. |
---|
191 | |
---|
192 | Awc/.svn/entries this_dir depth-empty |
---|
193 | ... |
---|
194 | Awc/B/.svn/entries this_dir depth-infinity |
---|
195 | ... |
---|
196 | |
---|
197 | svn up --set-depth=infinity Awc/D |
---|
198 | |
---|
199 | Pull everything into Awc/D, resulting in a subdirectory that is |
---|
200 | just as if it had been pulled in with no --depth flag at all. |
---|
201 | |
---|
202 | Awc/.svn/entries this_dir depth-empty |
---|
203 | B |
---|
204 | D |
---|
205 | Awc/D/.svn/entries this_dir depth-infinity |
---|
206 | ... |
---|
207 | ... |
---|
208 | |
---|
209 | ######################################################################## |
---|
210 | # # |
---|
211 | # ############################################################ # |
---|
212 | # ### ### # |
---|
213 | # ### THIS IS NOT YET FULLY IMPLEMENTED, SEE ISSUE #2843 ### # |
---|
214 | # ### ### # |
---|
215 | # ############################################################ # |
---|
216 | # # |
---|
217 | # svn up --set-depth=empty Awc/B/E # |
---|
218 | # # |
---|
219 | # Remove everything under E, but leave E as an empty directory # |
---|
220 | # since B is depth-infinity. # |
---|
221 | # # |
---|
222 | # Awc/.svn/entries this_dir depth-infinity # |
---|
223 | # B # |
---|
224 | # D # |
---|
225 | # Awc/B/.svn/entries this_dir depth-infinity # |
---|
226 | # ... # |
---|
227 | # Awc/B/E/.svn/entries this_dir depth-empty # |
---|
228 | # ... # |
---|
229 | # # |
---|
230 | # svn up --set-depth=empty Awc/D # |
---|
231 | # # |
---|
232 | # Remove everything under D, and D itself since A is depth-empty. # |
---|
233 | # # |
---|
234 | # Awc/.svn/entries this_dir depth-empty # |
---|
235 | # B # |
---|
236 | # # |
---|
237 | # svn up Awc/D # |
---|
238 | # # |
---|
239 | # Bring D back at depth-infinity. # |
---|
240 | # # |
---|
241 | # Awc/.svn/entries this_dir depth-empty # |
---|
242 | # ... # |
---|
243 | # Awc/D/.svn/entries this_dir depth-infinity # |
---|
244 | # ... # |
---|
245 | # ... # |
---|
246 | # # |
---|
247 | # svn up --set-depth=immediates Awc # |
---|
248 | # # |
---|
249 | # Bring in everything that's missing (C/ and mu) and empty all # |
---|
250 | # subdirectories (and set their this_dir to depth-empty). # |
---|
251 | # # |
---|
252 | # Awc/.svn/entries this_dir depth-immediates # |
---|
253 | # B # |
---|
254 | # C # |
---|
255 | # Awc/B/.svn/entries this_dir depth-empty # |
---|
256 | # Awc/C/.svn/entries this_dir depth-empty # |
---|
257 | # ... # |
---|
258 | # # |
---|
259 | # svn up --set-depth=files Awc # |
---|
260 | # # |
---|
261 | # Remove every subdirectories under Awc. but leave the files. # |
---|
262 | # # |
---|
263 | # Awc/.svn/entries this_dir depth-files # |
---|
264 | # # |
---|
265 | ######################################################################## |
---|
266 | |
---|
267 | |
---|
268 | 4. Implementation Strategy |
---|
269 | ========================== |
---|
270 | |
---|
271 | It would be nice if all this could be accomplished with just simple |
---|
272 | tweaks to how we drive the update reporter (svn_ra_reporter2_t). |
---|
273 | However, it's not that easy. |
---|
274 | |
---|
275 | Handling 'checkout --depth=empty' would be easy. It should get us |
---|
276 | an empty directory at depth-empty, with no files and no subdirs, |
---|
277 | and if we just report it as at HEAD every time, the server will |
---|
278 | never send updates down (hmmm, this could be a problem for getting |
---|
279 | dir property updates, though). Then any files or subdirs we have |
---|
280 | explicitly included we can just report at their respective |
---|
281 | revisions, and get proper updates; at least that'll work for the |
---|
282 | depth infinity ones. |
---|
283 | |
---|
284 | But consider 'checkout --depth=immediates'. The desired state is a |
---|
285 | depth-immediates directory D, with all files up-to-date, and with |
---|
286 | skeleton subdirs at depth empty. Plain updates should preserve this |
---|
287 | state of affairs. |
---|
288 | |
---|
289 | If we report D as at its BASE revision, files at their BASE |
---|
290 | revisions, and subdirs at HEAD, then: |
---|
291 | |
---|
292 | - When new files appear in the repos, they'll get sent down (good) |
---|
293 | - When new subdirs appear, they'll get sent down in full (bad) |
---|
294 | |
---|
295 | But if we don't report subdirs as at HEAD, then the server will try to |
---|
296 | update them (bad). And if we report D at HEAD, then the working copy |
---|
297 | won't receive new files that have appeared in the repository since D's |
---|
298 | BASE revision (note that we *can* get updates for files we already |
---|
299 | have, though, by continuing to report them at their respective BASEs). |
---|
300 | |
---|
301 | The same logic applies to subdirectories at depth-files or |
---|
302 | depth-immediates. |
---|
303 | |
---|
304 | So, for efficient depth handling, the client directly reports the |
---|
305 | desired depth to the server; i.e., we extend the RA protocol. |
---|
306 | |
---|
307 | Meanwhile, legacy servers will send back a bunch of information the |
---|
308 | client doesn't want, and the client just ignores it, and the user |
---|
309 | never knows except for the fact that everything seems slow (but |
---|
310 | once their servers are upgraded, it'll speed up). |
---|
311 | |
---|
312 | 5. Compatibility Matters |
---|
313 | ======================== |
---|
314 | |
---|
315 | This feature introduces two new concepts into the RA protocol which |
---|
316 | will not be understood by older servers: |
---|
317 | |
---|
318 | * Reported Depths -- the depths associated with individual paths |
---|
319 | included by the client in the description (via the |
---|
320 | svn_ra_reporter_t) of its working copy state. |
---|
321 | |
---|
322 | * Requested Depth -- the single depth value used to limit the |
---|
323 | scope of the server's response to the client. |
---|
324 | |
---|
325 | As such, it's useful to understand how these concepts will be |
---|
326 | handled across the compatibility matrix of depth-aware and |
---|
327 | non-depth-aware clients and servers. |
---|
328 | |
---|
329 | NOTE: in the sections below, it is not necessarily that case that a |
---|
330 | value or state which is said to be "transmitted" literally has a |
---|
331 | presence in the RA protocol. Some such bits of state have default |
---|
332 | values in the protocol and can therefore be effectively transmitted |
---|
333 | while not literally identifiable in a network trace of the |
---|
334 | client-server traffic. |
---|
335 | |
---|
336 | Depth-aware Clients (DACs) |
---|
337 | |
---|
338 | DACs will transmit reported depths (with "infinity" as the |
---|
339 | default) and will transmit a requested depth (with "unknown" as |
---|
340 | the default). They will also -- for the sake of older, |
---|
341 | non-depth-aware servers (NDASs) -- transmit a requested recurse |
---|
342 | value derived from the requested depth: |
---|
343 | |
---|
344 | depth recurse |
---|
345 | ----- ------- |
---|
346 | empty no |
---|
347 | files no |
---|
348 | unknown yes |
---|
349 | immediates yes |
---|
350 | infinity yes |
---|
351 | |
---|
352 | When speaking to an NDAS, the requested recurse value is the |
---|
353 | only thing the server understands , but is obviously more |
---|
354 | "grainy" than the requested depth concept. The DAC, therefore, |
---|
355 | must filter out any additional, unwanted data that the server |
---|
356 | transmits in its response. (This filtering will happen in the |
---|
357 | RA implementation itself so the RA APIs behave as expected |
---|
358 | regardless of the server's pedigree.) |
---|
359 | |
---|
360 | When speaking to a depth-aware server (DAS), the requested |
---|
361 | recurse value is ignored. A requested depth of "unknown" means |
---|
362 | "only send information about the stuff in my report, |
---|
363 | depth-aware-ily". Other requested depth values are honored by |
---|
364 | the server properly, and the DAC must handle the transformation |
---|
365 | of any working copy depths from their pre-update to their |
---|
366 | post-update depths and content as described in `3. Examples'. |
---|
367 | |
---|
368 | Non-depth-aware Clients (NDACs) |
---|
369 | |
---|
370 | NDACs will never transmit reported depths and never transmit a |
---|
371 | requested depth. But they will transmit a requested recurse |
---|
372 | value (either "yes" or "no", with "yes" being the default). (A |
---|
373 | DAS uses the presence of a requested depth in the actual protocol |
---|
374 | to distinguish DACs from NDACs, and knows to ignore the |
---|
375 | requested recurse value transmitted by a DAC.) |
---|
376 | |
---|
377 | When speaking to an NDAS, what happens happens. It's the past, |
---|
378 | man -- you don't get to define the interaction this late in the |
---|
379 | game! |
---|
380 | |
---|
381 | When speaking to a DAS, the not-reported depths are treated like |
---|
382 | reported depths of "infinity", and the reported recurse values |
---|
383 | "yes" and "no" map to depths of "infinity" and "files", |
---|
384 | respectively. |
---|
385 | |
---|
386 | 6. API Changes |
---|
387 | ============== |
---|
388 | |
---|
389 | A new enum type 'svn_depth_t depth' is defined in svn_types.h. |
---|
390 | Both client and server side now understand the concept of depth, |
---|
391 | and the basic update use cases handle depth. See depth_tests.py. |
---|
392 | |
---|
393 | On the client side, most of the svn_client.h interfaces that |
---|
394 | formerly took 'svn_boolean_t recurse' have been revved and their |
---|
395 | successors take 'svn_depth_t depth' instead. Each old API now |
---|
396 | documents how it converts 'recurse' to 'depth'. |
---|
397 | |
---|
398 | Some of this recurse-becomes-depth change has propagated down into |
---|
399 | libsvn_wc, which now stores a depth field in svn_wc_entry_t (and |
---|
400 | therefore in .svn/entries). The update reporter knows to report |
---|
401 | differing depths to the server, in the same way it already reports |
---|
402 | differing revisions. In other words, take the concept of "mixed |
---|
403 | revision" working copies and extend it to "mixed depth" working |
---|
404 | copies. |
---|
405 | |
---|
406 | On the server side, most of the significant changes are in |
---|
407 | libsvn_repos/reporter.c. The code that receives update reports now |
---|
408 | receives notice of paths that have different depths from their |
---|
409 | parent, and of course the overall update operation has a global |
---|
410 | depth, which applies except when restricted by some shallower local |
---|
411 | depth for a given path. |
---|
412 | |
---|
413 | The RA code on both sides knows how to send and receive depths; the |
---|
414 | relevant svn_ra_* APIs now take depth arguments, which sometimes |
---|
415 | supersede older 'recurse' booleans. In these cases, the RA layer |
---|
416 | does the usual compatibility dance: receiving "recurse=FALSE" from |
---|
417 | an older client causes the server to behave as if "depth=files" |
---|
418 | had been transmitted. |
---|
419 | |
---|
420 | 7. Work Remaining |
---|
421 | ================= |
---|
422 | |
---|
423 | The list of outstanding issues is shown by this issue tracker query |
---|
424 | (showing Summary fields that start with "[sparse-directories]"): |
---|
425 | |
---|
426 | <http://subversion.tigris.org/issues/buglist.cgi?component=subversion&issue_status=NEW&issue_status=STARTED&issue_status=REOPENED&short_desc=%5Bsparse-directories%5D&short_desc_type=casesubstring> |
---|