-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
645 lines (461 loc) · 31.9 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
<!DOCTYPE html>
<html class="no-js" lang="en-us" xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en-us">
<head>
<meta name="generator" content="Hugo 0.80.0" />
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="description" content="Ian's Blog">
<meta name="author" content="">
<meta name="keyword" content="">
<link rel="shortcut icon" href="/favicon.ico">
<title>Ian's Blog</title>
<link rel="stylesheet" href="https://bigredmachine.github.io/css/theme/journal.css">
<link rel="stylesheet" href="https://bigredmachine.github.io/css/font-awesome.min.css">
<link rel="stylesheet" href="https://bigredmachine.github.io/css/style.css">
<script src="https://bigredmachine.github.io/js/jquery.min-2.1.4.js"></script>
<script src="https://bigredmachine.github.io/js/bootstrap.min-3.3.5.js"></script>
<link href="https://bigredmachine.github.io/index.xml" rel="alternate" type="application/rss+xml" title="Ian's Blog" />
</head>
<body lang="en">
<div class="container">
<div class="row">
<div class="navbar navbar-default navbar-inverse" role="navigation">
<div class="navbar-header">
<a class="navbar-brand" href="https://bigredmachine.github.io">Ian's Blog</a>
</div>
<div class="navbar-collapse collapse navbar-responsive-collapse">
<ul class="nav navbar-nav navbar-right">
<li><a href="https://bigredmachine.github.io">Home</a></li>
<li><a href="https://bigredmachine.github.io/page/about/">About</a></li>
<li><a href="https://bigredmachine.github.io/post/">Blog</a></li>
</ul>
</div>
</div>
</div>
</div>
<div class="container">
<div class="row">
<div class="col-md-offset-1 col-md-10">
<h3><a href="https://bigredmachine.github.io/post/session-state-and-xdb-tracking-no-longer-required-for-rendering-personalisation/">Session state and Xdb tracking no longer required for rendering personalisation</a></h3>
<span class="label label-primary">Sun Feb 14, 2021</span> in
<a href="/tags/development">Development</a>
,
<a href="/tags/sitecore">Sitecore</a>
using tags
<a href="/tags/sitecore">Sitecore</a>
,
<a href="/tags/hotfix">Hotfix</a>
,
<a href="/tags/sessionstate">SessionState</a>
</small>
</div>
</div>
<div class="row">
<div class="col-md-offset-1 col-md-10">
<br>
<h2 id="happy-valentines-day">Happy Valentine’s day</h2>
<p> </p>
<picture>
<source
media="(min-width: 650px)"
srcset="/post/session-state-and-xdb-tracking-no-longer-required-for-rendering-personalisation/images/iheart_hua7cb347cab19114bde7edf1f069cf370_21611_495x0_resize_q100_box.jpg">
<source
media="(min-width: 465px)"
srcset="/post/session-state-and-xdb-tracking-no-longer-required-for-rendering-personalisation/images/iheart_hua7cb347cab19114bde7edf1f069cf370_21611_465x0_resize_q100_box.jpg">
<img
src="/post/session-state-and-xdb-tracking-no-longer-required-for-rendering-personalisation/images/iheart_hua7cb347cab19114bde7edf1f069cf370_21611_300x0_resize_q100_box.jpg"
alt="I heart Sitecore Support">
</picture>
<p> </p>
<h3 id="background">Background</h3>
<p>So for quite sometime in order to be able to personalise a rendering on Content Delivery you’ve had to turn on Xdb tracking, even if you don’t want to use Xdb. <br />
And by turning on Xdb Tracking, you have to turn on ASP.NET Session State, or it blows up.</p>
<ul>
<li>“Xdb.Tracking.Enabled” = true</li>
<li>“Xdb.Endabled” = false</li>
<li>ASP.NET Seession State - Enabled</li>
</ul>
<h3 id="personalisation-without-xdb">Personalisation without Xdb?</h3>
<p>What can you personalise with if you aren’t using Xdb?
Well certainly you are going to be limited what you can personalise on, but you can still personalise on</p>
<ul>
<li>Request Url</li>
<li>Request Headers</li>
<li>Request Cookies</li>
<li>Geo Location of Request</li>
</ul>
<p>Think anything in the current request, but not their History like you would if had an Xdb profile, or current browsing session for previous pages visited available with session state.</p>
<h3 id="whats-the-problem-with-turning-on-session-state-and-xdb-tracking">What’s the problem with turning on Session State and Xdb Tracking</h3>
<p>Well other than having to enable features that aren’t required, and it not being tidy.</p>
<p>Both of these two features cause cookies to be issued.</p>
<ul>
<li>SC_ANALYTCS_GLOBAL_COOKIE</li>
<li>ASP.NET_SessionId</li>
</ul>
<p>If you want to cache the response in a CDN then you can’t emit these request specific cookies at origin, the CDN won’t cache these responses. (Maybe with some custom logic to strip them?)</p>
<p>And trying to remove these cookies from being sent before the ASP.NET pipeline has flushed the response headers (after which can’t remove the headers as the client has already been sent the headers), can be quite tricky,
especially for the ASP.NET Session State cookie which gets added to the response when the ASP.NET Session State object is accessed if it doesn’t already exist.</p>
<p>So trying to find the sweet spot in the ASP.NET pipeline when Sitecore has finished accessing the Session State object, and before ASP.NET has flushed the response headers, can be quite tricky.
Or actually impossible to guarantee 100% of the time.</p>
<h3 id="solution---hotfix-for-sitecore-93">Solution - Hotfix for Sitecore 9.3</h3>
<p>Sitecore Hotfix 448700 for Sitecore 9.3 fixes this.</p>
<p>As I understand it this issue might have been fixed in Sitecore 10 possibly, and has been backported, based on conversations, but I’ve not tested if this is fixed in Sitecore 10.</p>
<p>With this hotfix, gone is the need for Session State on Content Delivery and Xdb.Tracking.Enabled. <br />
Gone is the need for the work arounds to remove these cookies from the response before they are sent. <br />
And can still have renderings personalised based on the current request.</p>
<p> </p>
<picture>
<source
media="(min-width: 650px)"
srcset="/post/session-state-and-xdb-tracking-no-longer-required-for-rendering-personalisation/images/cake_hueaac250c617d8a711f7ccc022cea4f61_50947_642x0_resize_q100_box.jpg">
<source
media="(min-width: 465px)"
srcset="/post/session-state-and-xdb-tracking-no-longer-required-for-rendering-personalisation/images/cake_hueaac250c617d8a711f7ccc022cea4f61_50947_465x0_resize_q100_box.jpg">
<img
src="/post/session-state-and-xdb-tracking-no-longer-required-for-rendering-personalisation/images/cake_hueaac250c617d8a711f7ccc022cea4f61_50947_300x0_resize_q100_box.jpg"
alt="Can have cake and eat it">
</picture>
<p> </p>
</div>
</div>
<div class="row">
<div class="col-md-12">
<hr>
</div>
</div>
<div class="row">
<div class="col-md-offset-1 col-md-10">
<h3><a href="https://bigredmachine.github.io/post/sitecore-log4net-lock-up/">Sitecore Log4Net Lockup - Hotfix</a></h3>
<span class="label label-primary">Sun Jan 31, 2021</span> in
<a href="/tags/development">Development</a>
,
<a href="/tags/sitecore">Sitecore</a>
using tags
<a href="/tags/log4net">Log4Net</a>
,
<a href="/tags/sitecore">Sitecore</a>
,
<a href="/tags/hotfix">Hotfix</a>
,
<a href="/tags/hang">Hang</a>
</small>
</div>
</div>
<div class="row">
<div class="col-md-offset-1 col-md-10">
<br>
<h2 id="sitecore-log4net-lockup---hotfix--dont-share-appenders">Sitecore Log4Net Lockup - Hotfix & Don’t Share Appenders</h2>
<p>This post is about a very rare elusive bug that caused the affected Sitecore instance to freeze up, as well not log anything which gave a clue what the problem was.</p>
<p>It was a combination of a bug in the Sitecore Log4Net code and additional Log4Net configuration that had been added (not OOTB) that caused Sitecore to lock up.</p>
<picture>
<source
media="(min-width: 650px)"
srcset="/post/sitecore-log4net-lock-up/images/padlock_hud18f49ec515f36941c728dfeefc35a24_662916_650x0_resize_q100_box.jpg">
<source
media="(min-width: 465px)"
srcset="/post/sitecore-log4net-lock-up/images/padlock_hud18f49ec515f36941c728dfeefc35a24_662916_465x0_resize_q100_box.jpg">
<img
src="/post/sitecore-log4net-lock-up/images/padlock_hud18f49ec515f36941c728dfeefc35a24_662916_300x0_resize_q100_box.jpg"
alt="Lock">
</picture>
<p>If you haven’t seen this issue, where a Sitecore instance locks up and you needed to restart IIS/Server, you probably won’t need this hotfix, but hopefully will get this hotfix rolled into a future version of Sitecore, so you’ll get the benefit then of hopefully never seeing this bug. <br />
The Log4Net configuration recommendations are worth following though should you add any bespoke configuration.</p>
<p>As I mentioned it was a rare bug, and could go months without seeing the issue surface.</p>
<p>The affected Sitecore instance performed publishing and indexing operations, so was quite busy with lots of threads. <br />
This issue has been seen in multiple Sitecore versions, up to at least Sitecore 9.3.</p>
<p>To get to the bottom of this elusive bug required a memory dump of the affected server. Sometimes restoring service is more important so not always possible to get a memory dump.</p>
<p>Even once that was done, required diagnostics to be added to work out what was causing the lock up/what was null, and more waiting for the issue to re-occur.</p>
<h3 id="what-was-locking-upwhat-was-null">What was locking up/What was null?</h3>
<p>Looking in the Sitecore.Logging.dll <br /></p>
<p>And the “PatternParser” class, you can see if you pass in any of %C, %F, %L, %M, %l in your log4net conversionPattern configuration, it will call either the “ClassNamePatternConverter” or the “LocationPatternConverter”:</p>
<picture>
<source
media="(min-width: 650px)"
srcset="/post/sitecore-log4net-lock-up/images/PatternParser_hu8aac3e27f9f65af552070b966547a495_37309_650x0_resize_q100_box_2.PNG">
<source
media="(min-width: 465px)"
srcset="/post/sitecore-log4net-lock-up/images/PatternParser_hu8aac3e27f9f65af552070b966547a495_37309_465x0_resize_q100_box_2.PNG">
<img
src="/post/sitecore-log4net-lock-up/images/PatternParser_hu8aac3e27f9f65af552070b966547a495_37309_300x0_resize_q100_box_2.PNG"
alt="Pattern Parser">
</picture>
<p>Both of these then make calls to get the Location information, which constructs a new “LocationInfo” instance.</p>
<picture>
<source
media="(min-width: 650px)"
srcset="/post/sitecore-log4net-lock-up/images/LocationInformation_huc911b9f510ab5985018c99d64b4502eb_14562_650x0_resize_q100_box_2.PNG">
<source
media="(min-width: 465px)"
srcset="/post/sitecore-log4net-lock-up/images/LocationInformation_huc911b9f510ab5985018c99d64b4502eb_14562_465x0_resize_q100_box_2.PNG">
<img
src="/post/sitecore-log4net-lock-up/images/LocationInformation_huc911b9f510ab5985018c99d64b4502eb_14562_300x0_resize_q100_box_2.PNG"
alt="Location Information">
</picture>
<p>The constructor of the “LocationInfo” loops through the stack trace.</p>
<picture>
<source
media="(min-width: 650px)"
srcset="/post/sitecore-log4net-lock-up/images/DeclaringType_hu5629a8b18b739bf4b270958ed048c863_52098_650x0_resize_q100_box_2.PNG">
<source
media="(min-width: 465px)"
srcset="/post/sitecore-log4net-lock-up/images/DeclaringType_hu5629a8b18b739bf4b270958ed048c863_52098_465x0_resize_q100_box_2.PNG">
<img
src="/post/sitecore-log4net-lock-up/images/DeclaringType_hu5629a8b18b739bf4b270958ed048c863_52098_300x0_resize_q100_box_2.PNG"
alt="Declaring Type">
</picture>
<p>This code assumes that the DeclaringType is always going to be available, however it is possible that the stackFrame.GetMethod().DeclaringType might be null for lambda, which it was in our cases.</p>
<h3 id="possible-workaround">Possible workaround</h3>
<p>One possible workaround before getting a hotfix would have been to not pass in these patterns which were causing the issues. However having the class name and location info of where an issue occurred is useful information.</p>
<h3 id="dont-share-appenders">Don’t share appenders</h3>
<p>So Sitecore/Log4Net was failing to log when the stack trace had a Lambda which had a null DeclaringType. <br />
Why was this causing Sitecore to hang? <br />
Shouldn’t logging fail safe?</p>
<p>Well when a logger fails, it tries to log the issue to the root logger.</p>
<p>Our root logger was setup to also log to the same appender instance, also with the same ConversionPattern.</p>
<p>Due to the shared appender between loggers it was possible for deadlock to occur.</p>
<p>Setup/Key Facts</p>
<ol>
<li>RootLogger and PublishingLogger use the same appender instance</li>
<li>DoAppend locks the appender which invoked it</li>
<li>CallAppenders locks the logger which invoked it.</li>
</ol>
<p>Scenario</p>
<ol>
<li>Publishing Logger call CallAppenders and DoAppend => gets the lock of Appender instance and PublishingLoggerObject</li>
<li>RootLogger call CallAppeners and locks RootLoggerObject</li>
<li>RootLogger calls DoAppender for Appender and tries to get the lock of the Appender Instance. But is already locked by PublshingLogger so has to wait until finishes.</li>
<li>Whilst the PublishingLogger is trying to log, it gets an exception in LocationInfo, and so calls RootLogger to log the exception.</li>
<li>RootLogger calls CallAppender and tries to get the lock of RootLoggerObject but is already locked</li>
</ol>
<p>Deadlock</p>
<picture>
<source
media="(min-width: 650px)"
srcset="/post/sitecore-log4net-lock-up/images/deadlock_hu45f59038fbd33db2bad2749093e160b8_104660_500x0_resize_q100_box.jpg">
<source
media="(min-width: 465px)"
srcset="/post/sitecore-log4net-lock-up/images/deadlock_hu45f59038fbd33db2bad2749093e160b8_104660_465x0_resize_q100_box.jpg">
<img
src="/post/sitecore-log4net-lock-up/images/deadlock_hu45f59038fbd33db2bad2749093e160b8_104660_300x0_resize_q100_box.jpg"
alt="Deadlock">
</picture>
<h3 id="azurefallbackappender">AzureFallbackAppender</h3>
<p>Looking at
App_Config\Sitecore\ExperienceContentManagement.Administration\Sitecore.ExperienceContentManagement.Administration.config</p>
<p>The AzureFallbackAppender is setup the same, for all the loggers to share the same appender.</p>
<p>“AzureFallbackAppender was designed specifically for Sitcore Support Package generator and it allowed us to collect the log files while generating the package. It was decided that only one AzureFallbackAppender remains for all kinds of logs, because of performance reasons.”</p>
<p>So whilst this isn’t recommended, the AzureFallbackAppender is a limited specific scenario, and issues haven’t been encountered with this.</p>
<h3 id="application-insights-paas-appender">Application Insights PAAS appender</h3>
<p>Looking at the file in the Azure Websites project:
App_Config\Sitecore\Azure\Sitecore.Cloud.ApplicationInsights.config</p>
<p>can see has the patch swaps out the default appends for the ApplicationInsights Log4NetAppender, so can see has a separate appender per logger as recommended.</p>
<p>So following the recommended practice.</p>
<h3 id="summary">Summary</h3>
<ul>
<li>DeclaringType can be null for Lambdas</li>
<li>Check for nulls</li>
<li>Don’t share Log4Net appenders between loggers</li>
</ul>
<h3 id="sitecore-hotfix-reference-number">Sitecore hotfix reference number</h3>
<p>To monitor the progress of the related issue, please use reference number [437603]</p>
</div>
</div>
<div class="row">
<div class="col-md-12">
<hr>
</div>
</div>
<div class="row">
<div class="col-md-offset-1 col-md-10">
<h3><a href="https://bigredmachine.github.io/post/incremental-publishing-edge-case/">Sitecore Incremental Publishing Edge Case</a></h3>
<span class="label label-primary">Thu Aug 13, 2020</span> in
<a href="/tags/publishing">Publishing</a>
,
<a href="/tags/sitecore">Sitecore</a>
using tags
<a href="/tags/publishing">Publishing</a>
,
<a href="/tags/sitecore">Sitecore</a>
</small>
</div>
</div>
<div class="row">
<div class="col-md-offset-1 col-md-10">
<br>
<h2 id="what-kind-of-publishing-is-this-post-about">What kind of publishing is this post about?</h2>
<p>This post is about Incremental Publishing using the out of the box publishing mechanism, not the new Sitecore Publishing Service. There <a href="https://mikael.com/2019/07/learnings-from-a-year-of-implementing-sitecore-publishing-service/">maybe reasons why you can’t use SPS just yet</a> for your project.</p>
<p>Again if you are using a Smart or Full site publish this article won’t apply.
But it’s <a href="https://blog.rauljimenez.co.uk/sitecore-publishing-tips/">recommended to setup a scheduled incremental publish</a>.</p>
<p>This post relates to Sitecore 9.3, not reviewed if applicable for Sitecore 10 yet.</p>
<p>Great, so I know this is about Sitecore OOTB Incremental Publishing, and why it’s recommended to use this if SPS isn’t a good fit, tell me more about this edge case.</p>
<h2 id="edge-case">Edge Case</h2>
<p>Disclaimer, this is an Incremental Publishing Edge Case issue where an item which should be published never gets published, most of the time you probably wouldn’t see this issue occur, and if you did spot something wasn’t published as it should, you’d just smart publish/republish the unpublished items & possibly their children to fix the issue.</p>
<p>You might possibly be thinking this could be a user error, something might not have been pushed through workflow, or not created until very recently (Always good to explore for technical issues before jumping to conclusions).</p>
<p>However if you have ruled out any user issues, and reviewed the logs and confirmed that the item should have been picked up in an Incremental Publish, you need to start looking for the Ghost in the Machine.</p>
<h2 id="ghost-in-the-machine">Ghost in the Machine</h2>
<p><img src="/post/incremental-publishing-edge-case/images/ghost.gif" alt="Ghost"></p>
<p>Working closely with Sitecore Support and with enough diagnostics data from the rare occurrences when this issue would rear it’s head, it was spotted that these items which wouldn’t get picked up by any incremental publish were saved around the time of the scheduled publishes.</p>
<p>Let’s explore what’s going on, and why an item saved very near an incremental publish operation on a rare occurrence might not get picked up.</p>
<h2 id="sql-column-specifications--rounding">Sql Column Specifications & Rounding</h2>
<p>The “Properties” table stores the Last Publish date as a “String” including the ticks.</p>
<p>The “PublishQueue” table “Date” column stores the date in a <a href="https://docs.microsoft.com/en-us/sql/t-sql/data-types/datetime-transact-sql?view=sql-server-ver15#rounding-of-datetime-fractional-second-precision">Sql “Datetime” column which isn’t as precise</a> and results in rounding.</p>
<p>As does the “Item” table “Created” and “Updated” columns which both use the same Sql “datetime” column, with the same rounding issue.</p>
<picture>
<source
media="(min-width: 650px)"
srcset="/post/incremental-publishing-edge-case/images/DatetimeRounding_hudedd1ac0b6b650abbd8018f2506def47_25522_650x0_resize_q100_box_2.PNG">
<source
media="(min-width: 465px)"
srcset="/post/incremental-publishing-edge-case/images/DatetimeRounding_hudedd1ac0b6b650abbd8018f2506def47_25522_465x0_resize_q100_box_2.PNG">
<img
src="/post/incremental-publishing-edge-case/images/DatetimeRounding_hudedd1ac0b6b650abbd8018f2506def47_25522_300x0_resize_q100_box_2.PNG"
alt="Datetime Rounding">
</picture>
<p>Due to these differences, say a publishing job was started with value From 14:00:00.017545Z, it would be rounded to 14:00:00.003ms, and would miss an item added at 14:00:00.000ms.</p>
<p>These millisecond round issues could cause an issue with an item not getting published, however that’s very small window for the issue to occur.</p>
<h2 id="what-happens-when-you-save-an-item">What happens when you save an item?</h2>
<p>When you save an item, the items statistics get updated with the timestamp from C#/.NET in memory before getting persisted to the Database.</p>
<p>So the latency to get the record with the timestamp into the database could certainly be an issue/factor.</p>
<p>The datetime value used defaults to not storing the ticks/milliseconds on the item, and just defaults to <code>datetime.ToString("yyyyMMddTHHmmss")</code></p>
<p>Truncating any milliseconds or ticks.
(See <code>ItemStatistics.UpdateRevision</code>, <code>DateUtil.IsoNow</code>, <code>DateUtil.ToIsoDate</code>)</p>
<p>So that could certainly be an issue, if saving an item towards the end of the Second (with a high millisecond value).</p>
<p>On save items are added to the <code>publishqueue</code>, with the date field values Updated, PublishDate, UnpublishDate, ValidFrom, ValidTo.
(See <code>DefaultPublishManager.DataEngine_SavedItem</code>, <code>DefaultPublishManager.AddToPublishQueue</code>, <code>DefaultPublishManager.GetActionDateFields</code>)</p>
<p>So that truncated second value gets copied across to the <code>publishqueue</code> table.</p>
<h2 id="example-of-an-edge-case">Example of an Edge Case</h2>
<p>If I save an item at <code>2020-08-13T18:00:00.9881779+00:00</code></p>
<p>It will get saved as being created/modified at <code>2020-08-13T18:00:00.000</code>,
and added to the publish queue</p>
<p>And I have an incremental publish kick off at 18:00, and picks up from the last publish 17:00:00.103-18:00:00.103 as a slight delay in starting.
This doesn’t pick up the item which hasn’t been created yet at 18:00:00.9881779.</p>
<p>And the future publish operation won’t pick it up either as will look for items between 18:00:00.103-19:00:00.103, when it’s value will be saved as 18:00:00.000ms.</p>
<h2 id="work-around">Work around</h2>
<p>A work around would be to fix the query to include records affected by this precision/latency issue, at the expense of reprocessing some items more than once.</p>
<p>To do this can customise <code>GetPublishQueueEntries</code> method on the <code>SqlDataProvider</code></p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-csharp" data-lang="csharp"><span style="color:#66d9ef">public</span> <span style="color:#66d9ef">override</span> List<PublishQueueEntry>GetPublishQueueEntries(DateTime <span style="color:#66d9ef">from</span>, DateTime to, CallContext context)
{
<span style="color:#66d9ef">from</span> = <span style="color:#66d9ef">from</span>.AddSecond(-<span style="color:#ae81ff">1</span>);
<span style="color:#66d9ef">return</span> <span style="color:#66d9ef">base</span>.GetPublishQueueEntries(<span style="color:#66d9ef">from</span>, to, context);
}
</code></pre></div><div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-xml" data-lang="xml"><span style="color:#f92672"><configuration</span> <span style="color:#a6e22e">xmlns:patch=</span><span style="color:#e6db74">"http://www.sitecore.net/xmlconfig/"</span> <span style="color:#a6e22e">xmlns:set=</span><span style="color:#e6db74">"http://www.sitecore.net/xmlconfig/set/"</span><span style="color:#f92672">></span>
<span style="color:#f92672"><sitecore></span>
...
<span style="color:#f92672"><dataProviders></span>
<span style="color:#f92672"><main></span>
<span style="color:#f92672"><patch:attribute</span> <span style="color:#a6e22e">name=</span><span style="color:#e6db74">"type"</span> <span style="color:#a6e22e">value=</span><span style="color:#e6db74">"YourNamespace.CustomSqlServerDataProvider, YourAssembly"</span> <span style="color:#f92672">/></span>
<span style="color:#f92672"></main></span>
<span style="color:#f92672"></dataProviders></span>
...
<span style="color:#f92672"></sitecore></span>
<span style="color:#f92672"></configuration></span>
</code></pre></div><h2 id="worst-case">Worst Case</h2>
<p>Item statistics get updated at 17:59:59.999ms in C#</p>
<p>Item gets saved as 17:59:59.000ms but some latency before added to the database so doesn’t get added until after 18:00:00.103ms
Item gets added to the publish queue with value 17:59:59.000ms</p>
<p>Publish Window by default queries for
<code>18:00:00.103-19:00:00.103</code></p>
<p>with the 1 second fix would query for
<code>17:59:59.103-19:00:00.103</code></p>
<p>Both of these would still miss the item at 17:59:59.000ms.</p>
<h2 id="improved-work-around">Improved Work Around</h2>
<p>To round the start time query down to 0 milliseconds, and 0 ticks, can add the following line <a href="https://stackoverflow.com/questions/1004698/how-to-truncate-milliseconds-off-of-a-net-datetime">Credit Stack Overflow</a>:</p>
<div class="highlight"><pre style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4"><code class="language-csharp" data-lang="csharp"><span style="color:#66d9ef">public</span> <span style="color:#66d9ef">override</span> List<PublishQueueEntry>GetPublishQueueEntries(DateTime <span style="color:#66d9ef">from</span>, DateTime to, CallContext context)
{
<span style="color:#66d9ef">from</span> = <span style="color:#66d9ef">from</span>.AddSecond(-<span style="color:#ae81ff">1</span>);
<span style="color:#66d9ef">from</span> = <span style="color:#66d9ef">from</span>.AddTicks(-(<span style="color:#66d9ef">from</span>.Ticks % TimeSpan.TicksPerSecond));
<span style="color:#66d9ef">return</span> <span style="color:#66d9ef">base</span>.GetPublishQueueEntries(<span style="color:#66d9ef">from</span>, to, context);
}
</code></pre></div><p>So the new query would pick up even this more extreme edge case, with the query
<code>17:59:59.000-19:00:00.103</code>.</p>
<h2 id="summary">Summary</h2>
<p>TL;DR; OOTB if you save an item at the same time an incremental publish kicks off, depending on the exact timing there is an edge case that it may never get published.</p>
<p>With a small work around in code you can ensure items saved around the time of a scheduled publish get picked up by the next incremental publish. At the expense of reprocessing some items/events.
However this workaround isn’t in the form of an official hotfix, and would require extensive testing. No guarantees. Only consider this if are experiencing the issues described, and of course speak with Sitecore Support first.</p>
<h2 id="do-i-need-this">Do I need this?</h2>
<p>Well firstly you have to be using incremental publishing and scheduled publishing.
And secondly unless you’ve spotted items not getting published, have enough editor activity that keep on encountering this issue on rare occasions, you probably don’t need it.</p>
<p>However if you are using incremental and scheduled publishing, and have spotted odd issues with items not getting published, and can’t explain them due to user error. This might help explain the Ghost in the machine and a work around. But don’t apply this fix if you don’t need it.</p>
<p>And if you do implement this fix you’ll have to test this customisation, as not an official hotfix, and no guarantees. Just a suggestion that this could help if suffering from this particular issue.
And if was implemented incorrectly could lead big problems, e.g. re-processing of events.
So you’ve been warned.</p>
<h2 id="longer-term">Longer term</h2>
<p>Longer term I’m sure SPS will further improve, and will be able to be used by more customers. At which point Smart Publishing becomes fast enough, we don’t need the old incremental publishing.</p>
<p>If there was to be a fix for incremental publishing, the window could be reduced by calculating and storing a last modified time at Sql Server level with millisecond/timestamp precision, rather than using C# and truncating down to the second level with the latency to insert it into the database.
As well as using the DateTime2 column format rather than DateTime with it’s millisecond rounding precision issues.
However, this is quite a schema change for an edge case, so the work around will suffice for those that are affected.</p>
<h2 id="sitecore-bug-tracking-reference">Sitecore Bug Tracking Reference</h2>
<p>Sitecore Bug Tracking Reference #397573.</p>
<h2 id="final-disclaimer">Final Disclaimer</h2>
<p>Don’t use this work around unless you are experiencing the issues described, and even then perform your own testing, no guarantees.</p>
<p>These code samples haven’t been tested, and just give the gist of the fix required.</p>
<p>Work with Sitecore Support to confirm you are experiencing the issue described, as this customisation has a cost to it, and could have negative implications to your solution even if implemented correctly, and even more negative implication if incorrectly implemented.</p>
</div>
</div>
<div class="row">
<div class="col-md-12">
<hr>
</div>
</div>
</div>
<div class="container">
<div class="row">
<div class="col-md-12">
<div class="text-center">
<ul class="pagination">
<li class="page-item">
<a href="/" class="page-link" aria-label="First"><span aria-hidden="true">««</span></a>
</li>
<li class="page-item disabled">
<a class="page-link" aria-label="Previous"><span aria-hidden="true">«</span></a>
</li>
<li class="page-item active">
<a class="page-link" href="/">1</a>
</li>
<li class="page-item">
<a class="page-link" href="/page/2/">2</a>
</li>
<li class="page-item">
<a class="page-link" href="/page/3/">3</a>
</li>
<li class="page-item disabled">
<span aria-hidden="true"> … </span>
</li>
<li class="page-item">
<a class="page-link" href="/page/6/">6</a>
</li>
<li class="page-item">
<a href="/page/2/" class="page-link" aria-label="Next"><span aria-hidden="true">»</span></a>
</li>
<li class="page-item">
<a href="/page/6/" class="page-link" aria-label="Last"><span aria-hidden="true">»»</span></a>
</li>
</ul>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<hr>
</div>
</div>
</div>
<div class="container">
<div class="row col-md-12">
<footer>
<div class="pull-left">
<p>
© 2015-2021 Ian Jones.
</p>
</div>
<div class="pull-right">
<a href="https://twitter.com/darkhorseian" target="_blank">
<i class="fa fa-twitter-square fa-2x"></i></a>
<a href="https://bigredmachine.github.io/index.xml" target="_blank">
<i class="fa fa-rss-square fa-2x"></i></a>
</div>
</footer>
</div>
</div>
</body>
</html>