Invalidating the Result Cache in Doctrine & Symfony2

I’m currently developing a large project using symfony2 and doctrine which makes heavy use of Doctrine’s CacheProvider cache classes. I’ve been looking around for an easy way of invalidating the result cache when those objects are updated in the database.

I was unable to find any existing way to do this, as far as I know you can enable the result cache using the useResultCache(true) method and you can make that result cache expire after a time period. I was looking for a way to automatically invalidate cache entries when the entity was updated.

You can do this in your controller by using this

This wasn’t ideal for me because I would have to had to paste this code for every cache key into multiple actions in every controller or make a custom method etc.

My solution was to create a CacheInvalidator service which plugs into the doctrine onflush event. It then looks at all of the entities that are going to be changed (inserted, updated or deleted) and clears the result cache based on a configuration array.

There are two other things you can check for that I am not using in my service, $uow->getScheduledCollectionDeletions() and $uow->getScheduledCollectionUpdates(). I don’t use this because right now my application does not update collections in this way.

Right now I hard coded the configuration array into the service, but this could use yml or xml.

The configuration array contains keys with the entity classes and the cache keys that should be invalidated when there is a match. Each cache key contains a cache string and a number of placeholders. These placeholders can be dynamically determined based on the entity that is being changed.

Here is a quick example:

Let’s say you have an entity called BlogPost:

Normally the field $user would be a foreign key to a User entity but that’s not important here. Assume you have this repository class:

The getPostsForUser method does exactly what it says, pass in the a user ID and all the posts for that user are returned. It also caches the results after the first run.

Now what happens when a BlogPost is inserted by this user? or an existing one is deleted. The cached results for this query need to be invalidated.

Add this to your services.yml:

And here is the code for the service:

Update: I added a few changes to the class after posting. In your configuration each cache ID has a ‘change’ key that can be insert, update, delete or any. This allows you to only clear the cache when an entity is deleted.

For example if I set ‘change’ to ‘delete’ only when the user object is deleted would that cache key be cleared.

In the example above, I want ‘change’ to be ‘any’ so that when a BlogPost entity is added, modified or deleted, the cache is cleared automatically.

There is some code smell here, you have to repeat the cache keys in the repository classes and in the configuration. I haven’t decided the best way around this, if anyone has any ideas; leave a comment.

Also currently there is no error checking on the method paths in the configuration.

The cache ID config can contain multiple entities and each entity can contain multiple cache keys to clear. Right now there is no way to share the same cache key among multiple entities.

Assume the user_id of this blog post is 1. This config will clear the cachekey posts_for_user_1. The value 1 is determined by calling $entity->getUser(). You can use dot notation if you need to call more methods. i.e

getUser.getId

would result in a call to $entity->getUser()->getId(). Note that you cannot use the short twig syntax user.id yet. As above make sure the method paths are correct, there is no error checking in place for this yet.

Don’t forget to set a result cache driver in config.yml, there are several available including memcache, memcached, redis and apc. ex:

I’d like to hear your feedback, is the better way to do this? I think that something like this should be in Doctrine or Symfony2 by default.

,

18 thoughts on “Invalidating the Result Cache in Doctrine & Symfony2

  • Eduardo says:

    Hi there!
    Great Post, I was searching something similar and I didn’t remeber the Doctrine Events could help this way. I like this solution!
    But I have a concern: wouldn’t this add too much overhead, the same that you are trying to avoid with cache?Where you able to test that trade of?

    • Greg Freeman says:

      It depends on your situation and how many items you are caching, if you were worried about cache slams, you could regenerate your cache using something like gearman so that cached items are not being created during browser requests and use a cache lock field. The above service could easily be split off into a background process.

  • Hi all, first, your post might be very useful to me for my resultCache invalidation,but, I am facing an issue (quit simple for you).

    I Implemented a class called \Groupe01\ORM\Cache\CacheEventsListener which declare quite the same method (I am trying to delete all NamedQueries associated with the entity when Flush Events is fired).

    Here is the code :

    public function onFlush(OnFlushEventArgs $eventArgs) {
    $logger = $this->getLogger(__METHOD__);
    if ($logger->isDebugEnabled()){
    $logger->debug(‘OnFlush raised !’);
    }
    $em = $eventArgs->getEntityManager();
    $uow = $em->getUnitOfWork();
    // $uow->getScheduledCollectionDeletions()
    // $uow->getScheduledCollectionUpdates()
    $scheduledEntityChanges = array(
    ‘insert’ => $uow->getScheduledEntityInsertions(),
    ‘update’ => $uow->getScheduledEntityUpdates(),
    ‘delete’ => $uow->getScheduledEntityDeletions()
    );
    $cacheEntries = array();
    $em = new EntityManager($conn, $config, $eventManager);
    foreach ($scheduledEntityChanges as $change => $entities) {
    foreach($entities as $entity) {
    $cacheEntityId = $entity->getCacheKey();
    if ($logger->isDebugEnabled()){
    $logger->debug(‘Treating entity : ‘.$entity->toArray(0));
    }
    if (! array_key_exists($cacheEntityId, $cacheEntries)) {
    $cacheEntries[$cacheEntityId][‘provider’] =
    CacheFactory::getCacheManagerInstance(ApplicationConfig::getConfigVar(EntityManagerFactory::DOCTRINE_CACHE_PROVIDER), $entity->getNamespace());
    }

    if ($em->getMetadataFactory()->getMetadataFor(get_class($entity))->hasNamedQueries()) {
    $cacheEntries[$cacheEntityId][‘resultCacheIds’] = array();
    $entityNamedQueries = $em->getClassMetadata(get_class($entity))->getNamedQueries();
    foreach ($entityNamedQueries as $namedQuery) {
    if ($logger->isDebugEnabled()){
    $logger->debug(‘Treating namedQuery : ‘.$namedQuery);
    }
    $cacheEntries[$cacheEntityId][‘resultCacheIds’][] = $entity->getClazzName().’_’.$name;
    }
    }

    }
    }
    if (count($cacheEntries) == 0) {
    return;
    }
    $cacheEntries = array_unique($cacheEntries);
    foreach ($cacheEntries as $entityKey => $cacheValues) {
    $cacheManager = $cacheValues[‘provider’];
    $cacheManager->delete($entityKey);
    if (array_key_exists(‘resultCacheIds’, $cacheValues)) {
    array_map(array($cacheManager, ‘delete’), $cacheValues[‘resultCacheIds’]);
    }
    }
    }

    I have a factory to get the right EntityManager associated to the good database : here is the entityManager instantiatin procedure :
    public static function getEntityManager($dbCode = null, $clazz) {
    $logger = self::getLogger(__METHOD__);
    if (empty($dbCode)) {
    $dbCode = ‘default’;
    }
    if (!isset(self::$_entityManagers[$dbCode])) {
    try {
    $configuration = self::getConfiguration();
    $eventManager = new EventManager();
    if (ApplicationConfig::getConfigVar(EntityManagerFactory::RESULT_CACHE_ENABLED)) {
    $eventManager->addEventListener(array(Events::postFlush), \Groupe01\ORM\Cache\CacheEventsListener::onFlush());
    }
    self::$_entityManagers[$dbCode] = EntityManager::create(ConnexionFactory::getConnexion($dbCode, $configuration), $configuration, $eventManager);
    self::$_entitiesDbCodeMap[$clazz] = $dbCode;
    } catch (InvalidArgumentException $ex) {
    $logger->fatal(‘Unable to initialize connection (W) ‘ . $dbName ? $dbName : “default”);
    return null;
    } catch (Exception $ex) {
    $logger->fatal(‘Unexpected exception for connection :’ . $dbName ? $dbName : “default”);
    return null;
    }
    }
    return self::$_entityManagers[$dbCode];
    }

    When testgin this code, I have this error :

    PHP Catchable fatal error: Argument 1 passed to Groupe01\ORM\Cache\CacheEventsListener::onFlush() must be an instance of Groupe01\ORM\Cache\OnFlushEventArgs, none given,

    Any Idea what I am doing wrong ?

    Thanxs a lot for your time and help !

    Marc

    • Greg Freeman says:

      Looks like you forget to include the namespace for OnFlushEventArgs, you need to add:

      use Doctrine\ORM\Event\OnFlushEventArgs;

      You can see from your error message, it is looking for the class inside the wrong namespace.

  • Javier Spagnoletti says:

    I added a basic comprobation to check valid entity keys (https://gist.github.com/phansys/5164736).

    I’ve tried to add the class keys and his config at each entity’s @PostLoad event, but i couldn’t because a recursive loop in stack.

    Something like:


    <?php
    namespace MyVendor\MyBundle\Entity;
    use Doctrine\ORM\Mapping as ORM;
    use Symfony\Component\Validator\Constraints as Assert;
    use Symfony\Bridge\Doctrine\Validator\Constraints\UniqueEntity;
    /**
    * @ORM\Entity(repositoryClass="MyVendor\MyBundle\Entity\MyEntityRepository")
    * @ORM\Table(name="my_entity")
    * @ORM\HasLifecycleCallbacks()
    */
    class MyEntity
    {
    //...

    /**
    *
    * @ORM\PostLoad()
    */
    public function addToCache()
    {
    CacheInvalidator::addEntityToCache($this);
    }
    }

    In CacheInvalidator.php:


    //...
    public static function addEntityToCache($entity)
    {
    self::$cacheIds[get_class($entity)] = $entity->getCacheConfig();
    }


    You’ve got any ideas?

  • Eric says:

    This works really well. Thank you for sharing!

    I modified it a bit by turning it into an abstract class with

    left to implement by subclasses.

  • solitario says:

    I believe you’re missing important line:

    array_map(array($resultCache, ‘delete’), $cacheIds);

    After that you should add:

    $resultCache->flushAll();

    Otherwise the cache won’t be cleared. Tested on PHP 5.6.6 & memcached 2.1.0

  • solitario says:

    Yeah you’re right Greg.
    I’ve installed phpMemcachedAdmin  and was able to see what actually was going on. (not very familiar with memcached – I used APC till PHP 5.5)
    I’ve moved to your approach. The only downside is that I need to set setResultCacheId and since I have many different Entities (over 10 with at 3-5 custom repository methods) it’s going to be crazy (especially when some queries have additional parameters).

    Turns out that memcache  now doesn’t have an option to search by prefix (it would be more easy to delete all cached keys by same prefix or pattern), or at least I wasn’t able to find out.

    Regards

  • […] I’m now invalidating the result cache using onFlush event (This is the only way I found and the article is too old back in 2012. Is there any better recent […]

  • […] I’m now invalidating the result cache using onFlush event (This is the only way I found and the article is too old back in 2012. Is there any better recent […]

  • Nico Phil says:

    Hello and thanks for your article,

    i am wondering how you can deal with relationships between entities.

    Let’s say we have comments on the posts.

    The whole repository’s method might be :

    Now, let’s say that admin can moderate the comments : if (s)he modifies a comment, the only way i found to invalidate the good cache is to define the cacheIds array for all entities involved in the relation. This could be pretty heavy if you have multi joints queries…

  • Lou says:

    Bit late to the party, but thanks for the code. It functions really well and saved me reinventing the wheel. Appreciate the article and code, thanks again.

  • Lost Packet says:

    This article is still going strong, as Lou said, saved reinventing the wheel.

    Excellent job works perfectly, this should be built in.

  • donny says:

    Sometimes symfony can return class name which have prefix “Proxies\__CG__\..” so it won’t flush our cache. To fix this replace

    with

    and should return real class. Whole script saves me a lot of time, but “bug above cost me 1-2 hours 😉

  • Sam Rogers says:

    Hi Greg,

    Thanks for the post I think this will help me out a lot!

    A couple modifications to what you have here in case you were interested.

    As the the things you need to cache grow I think managing 1 big array of cache keys will get unwieldy so I moved it to the entities themselves using an interface:

    Then in your entity in my case BlogLike:

     

    Finally change your logic here so it doesn’t need to parse the configuration it just retrieves the cache ids from the entitiy:

    Thanks again for writing this post the above is just my spin on it!

    Sam

     

     

     

     

     

     

Leave a Reply to Greg Freeman Cancel reply

Your email address will not be published. Required fields are marked *