Skip to content

NDB Pagestore implementation#1173

Merged
danbri merged 3 commits intomasterfrom
ndbpagestore
Jun 21, 2016
Merged

NDB Pagestore implementation#1173
danbri merged 3 commits intomasterfrom
ndbpagestore

Conversation

@RichardWallis
Copy link
Contributor

In response to issue (#1171)
Overview:

  • Previously created pages are held in a within instance memory cache that is shared between threads. This means that each page is recreated for each application instance. We previously noted that the live site spawned off dozens of instances to support the load,
  • This version uses the Cloud Datastore to hold cached pages.
  • The single NDB store is shared between all threads of all instances.
  • ​_Should_​ result in a lot less page creations, and more importantly you having to wait for a new instance to create you a page.
  • The NDB store is persistent so will keep the page store even when you reload/deploy a new version of the code/data.
  • Implemented code that detects this condition and deletes all pages in the store as it starts.
  • For early supporting there is a way to manually call this caheFlush function.

If successful a similar approach should be considered for the underlying RDF data.

@RichardWallis RichardWallis added the site tools + python code Infrastructural issues around schema.org site. Most can ignore this! label May 20, 2016
@RichardWallis RichardWallis self-assigned this May 20, 2016
Added json extenstion to etags for json pages
@RichardWallis
Copy link
Contributor Author

Superseded by #1213

danbri added a commit that referenced this pull request Jun 17, 2016
Ndbpagestore into Makemake

Copying notes from #1173 
In response to issue (#1171) 
Overview:

Previously created pages are held in a within instance memory cache that is shared between threads. This means that each page is recreated for each application instance. We previously noted that the live site spawned off dozens of instances to support the load,
This version uses the Cloud Datastore to hold cached pages.
The single NDB store is shared between all threads of all instances.
​Should​ result in a lot less page creations, and more importantly you having to wait for a new instance to create you a page.
The NDB store is persistent so will keep the page store even when you reload/deploy a new version of the code/data.
Implemented code that detects this condition and deletes all pages in the store as it starts.
For early supporting there is a way to manually call this caheFlush function.
@RichardWallis
Copy link
Contributor Author

Reopening to enable applying to master

@RichardWallis RichardWallis reopened this Jun 21, 2016
@danbri danbri merged commit c0505a0 into master Jun 21, 2016
@danbri
Copy link
Contributor

danbri commented Jun 21, 2016

Ok, 2nd time was successful. I had to hunt for the settings to enable datastore writes in schema.org version of the site. For reference that is found via https://cloud.google.com/appengine/docs/python/console/managing-datastore

@danbri danbri deleted the ndbpagestore branch August 10, 2016 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

site tools + python code Infrastructural issues around schema.org site. Most can ignore this!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants