Skip to content

Fix feed delta implementation.#831

Merged
philwareham merged 1 commit intotextpattern:devfrom
da2x:feature_delta_feeds
Nov 3, 2016
Merged

Fix feed delta implementation.#831
philwareham merged 1 commit intotextpattern:devfrom
da2x:feature_delta_feeds

Conversation

@da2x
Copy link
Copy Markdown
Contributor

@da2x da2x commented Nov 3, 2016

New implementation supports returning feed deltas (just the new entries) if the A-IM plus either the If-Modified-Since or If-None-Match headers to abide by feed caching best practices.

  • Always send a Vary header for feeds so caches will be variant aware.
  • Disable caching when Txp is not in production.
  • Don't lie about last modification dates.
  • Derive ETags from last modification timestamps.

Follow up from issue #811, obsoletes #826.

New implementation supports returning feed deltas (just the new entries)
if the A-IM plus either the If-Modified-Since or If-None-Match headers.

* Always send a Vary header for feeds so caches will be variant aware.
* Disable caching when Txp is not in production.
* Don't lie about last modification dates.
* Derive ETags from last modification timestamps.
@da2x
Copy link
Copy Markdown
Contributor Author

da2x commented Nov 3, 2016

Test instructions

curl -i -H "A-IM: feed" -H "If-Modified-Since: Thu, 03 Nov 2016 12:00:00 GMT" http://localhost/textpattern/atom

Tweak the date backward in time. The feed should only include entries modified/published after the datestamp. If no new entries are return, you should get no body message/feed and get a 304 response code.

@makss
Copy link
Copy Markdown
Contributor

makss commented Nov 3, 2016

It seems we are with you at the same time fix the same piece of code. ;-)
I'm still trying to optimize and synchronize code atom.php with rss.php. Wait a little bit and we will try to find the most optimal way.

BTW, someone actually uses "A-IM: feed" aka RFC-3229 aka Delta encoding for rss/atom feeds?

I absolutely do not see any reason to use it in such small amounts of data.

ps: The current implementation of this (A-IM) algorithm in Textpattern made in error. More than 10 years on it, nobody paid attention, most likely A-IM nobody uses.

@da2x
Copy link
Copy Markdown
Contributor Author

da2x commented Nov 3, 2016

Wait a little bit and we will try to find the most optimal way.

No reason not to merge this now, is there? Incremental improvements and then you can sort out the conflicts in your branch rather than having me sort them out in mine. 😉 This patch does reduce the number of differences between the two implementations. I actually thought I’d try to combine them into feed.php next. Just for fun. I guess both of us should have communicated better through the bug tracking system about our intentions.

BTW, someone actually uses "A-IM: feed" aka RFC-3229 aka Delta encoding for rss/atom feeds?

Clients: FeedHQ, Feed Notifier, Internet Explorer and Microsoft Outlook (Widows-RSS-Platform), Liferea, Miniflux, Newsblur, Newsbeuter, Vienna, and patches exist for even more. Most of them didn’t support it two weeks ago, but I’ve implemented it in all the still-maintained open source feed readers.

Servers: Textpattern, ExpressionEngine, WordPress (with a plugin), and OpenFire.

I absolutely do not see any reason to use it in such small amounts of data.

Mobile and Australia (pay-per-MB). Also, feeds can be configured to include hundreds of entries and website that publishes hundreds of entries per day would need to increase the number of entries in their feeds. Podcasts traditionally include the last year or two of episodes in their feeds. Plus, with most cheap hosting solutions webadmins will be more restricted by bandwidth constrains than the extra microsecond it takes to get feed deltas right. Full-length feeds is a thing that can be enabled in settings. Give and take a few of these conditions, and you’ll start wishing more feed producers and clients supported delta feeds.

@philwareham philwareham merged commit 4708dc5 into textpattern:dev Nov 3, 2016
@philwareham
Copy link
Copy Markdown
Member

Thanks, I've merged this so you can discuss between you best way to continue the improvements together. :)

@makss
Copy link
Copy Markdown
Contributor

makss commented Nov 3, 2016

I guess both of us should have communicated better through the bug tracking system about our intentions.

That was my miscalculation. I just watched the code and it turned out that I changed it. ;-)

Clients: FeedHQ, Feed Notifier, Internet Explorer and Microsoft Outlook (Widows-RSS-Platform), Liferea, Miniflux, Newsblur, Newsbeuter, Vienna, and patches exist for even more. Most of them didn’t support it two weeks ago, but I’ve implemented it in all the still-maintained open source feed readers.
.
Servers: Textpattern, ExpressionEngine, WordPress (with a plugin), and OpenFire.

I agree that these systems declare work with delta-feed. But how many of them can work out all the situation correctly?

Test cases:

  • Future posts
  • Mass edit: category, section, author

They do not work out. These test cases are applicable to other systems.

There are many nuances (as well, and other plugins that these changes may affect) and more likely to make mistakes, because of which the loss of positions in the Feed Reader may occur. I would prefer reliable operation of the core system.

Delta feed can be developed as a plugin, it will give more flexibility and greater speed of development. The plugin can already be put on existing sites, rather than wait for the next release of the core system.

@makss
Copy link
Copy Markdown
Contributor

makss commented Nov 3, 2016

@Aeyoun function handle_lastmod() has been broken. This function is not only used for feed, so I would not change it so drastically.

@da2x
Copy link
Copy Markdown
Contributor Author

da2x commented Nov 3, 2016

@makss, broken how? No one consumed that functions return values, and that is all I removed from it. The biggest change is that it now sets ETags as well as Last-Modified. It ensures compatibility with more clients, as not all clients implement If-Modified-Since (not all implement If-None-Match either, so that is why we still use both all over the web).

It does the same job as before: it still takes care of signaling cache freshness. It still makes sure the body message is empty by killing the processing after sending a 304.

Maybe I missed something, but I don’t believe that I broke it in any way.

@makss
Copy link
Copy Markdown
Contributor

makss commented Nov 4, 2016

Technical problems:

  • php 5.3
    Parse error: syntax error, unexpected '[' in /home/mhosts/txp46/txp46rss/textpattern/textpattern/lib/txplib_misc.php on line 4572
    line $inmd = ($hinm) ? base_convert(explode('-gzip', $hinm)[0], 32, 10) : 0;
  • no comments... :-(
       if ((isset($imsd) && $imsd >= $unix_ts) ||
             (isset($inmd) && $inmd >= $unix_ts)) {   return;
             log_hit('304');
             ...rest code....
       }

Please test your own code.

Logical problems:

  • handle_lastmod() function can use plugins for this purpose and is a return to the status of the array.
  • A function handle_lastmod() passes the entire output Textpattern and output plugins. Duplicate Last-Modified inETag can still be used in feeds, but it is not permissible for the rest of the content. For example in one of the plugins I use Etag like md5 ( 'my_base_content') it allows me to ignore minor changes on the page. The content has no time reference.

@da2x
Copy link
Copy Markdown
Contributor Author

da2x commented Nov 4, 2016

php 5.3 […] Please test your own code.

I didn’t test with ancient versions of PHP, that is correct. Looks like explode can’t be used like an array before after PHP 5.4. PHP versions older than 5.6 is unsupported (and 5.6 only has a a year left of its life) and aren’t getting security updates. I’ll find a workaround for this issue, but in the meantime you should update your PHP version.

no comments... :-(

Now you’re just being unfair. There where hardly any comments there before. I’ll sprinkle over some more.

use[d in] plugins

Ah. Good point. I’ll put the returns back.

@da2x
Copy link
Copy Markdown
Contributor Author

da2x commented Nov 4, 2016

Actually, I retract the first one. I’ll not bother working around an issue in an insecure and already two-year unmaintained version of PHP. I can put in a warning about running ancient versions of PHP somewhere instead, if that is desired.

@philwareham
Copy link
Copy Markdown
Member

The minimum PHP version for Textpattern is currently 5.3 so, although admittedly annoying, code should respect that. PHP 5.3 is still fairly widely used.

@da2x
Copy link
Copy Markdown
Contributor Author

da2x commented Nov 4, 2016

code should respect that.

Added to pull #832.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants