Blog

This post is kind of continuation of my last post on Inventory app with DRF in Django. As an enhancement, an approval workflow to moderate any updation/deletion/addition of a new model entry was to be put in place. Now, I did look up for ready made solutions and could find a few open source ones – Django Moderation was one of them. However, it had its limitation and was not compatible with the latest Django 3.

So, I decided to come-up with a custom solution utilizing the DRF framework as well. This post basically outlines the approach I took, the challenges I faced and how they were resolved.

To start with, below is the flow diagram of the approval workflow that has been implemented:

The approach followed to implement the above:

1. Create an abstract model which needs moderation
2. Create 2 models inheriting the abstract model – One acts as the primary model, and the other an approval model, where changes can be moderated and then saved to the primary model based on actions taken by moderator
3. Instead of using Django admin site for data entry, create a form (template) to Add/Update data – Used DRF Forms for this as all permission and field serialization is handled by DRF – so effective DRY principal!
4. Create a separate template for the Approval model, from where entries can be moderated.

Whenever an entry is added/updated, it is first saved in the approval model. On successful moderation, the entry is saved in the primary model. Sounds simple right? 😉 Well, let’s find that out.

Now, a lot of templating work can be prevented in-case you go with the out of the box Django admin site, however I would suggest to spend some time to have templates designed for these actions which would give a more professional look to the site and also allow granular customizations. (especially in the UI)

Let’s have a look at the model:

# Abstract Model
class AbstractServer(models.Model):
    component = models.ForeignKey(Component, blank=False, null=False, on_delete=models.CASCADE)
    name = models.CharField(max_length=50, unique=True, blank=False, null=False)
    ip = models.GenericIPAddressField(unique=True, blank=False, null=False)
    dc = models.ForeignKey(Dc, blank=False, null=False, on_delete=models.CASCADE)
    environment = models.ForeignKey(Environment, blank=False, null=False, on_delete=models.CASCADE)
    type = models.ForeignKey(Type, blank=True, null=True, on_delete=models.CASCADE)
    state = models.CharField(
        max_length=10,
        choices=STATES,
        blank=False
    )
    tags = models.ManyToManyField(Tag, blank=True)
    group = models.ForeignKey(Group, blank=False, null=False, on_delete=models.CASCADE)
    description = models.TextField(blank=True, null=True)
    requestor = models.ForeignKey(User, blank=False, null=False, default=1, on_delete=models.CASCADE)
    
    class Meta:
        abstract = True

# Primary Model (inherited from abstract model)
class Server(AbstractServer):
    changed_by = models.ForeignKey(User, blank=False, null=False, on_delete=models.CASCADE, related_name='server_changed_by')
    history = HistoricalRecords()

    @property
    def _history_user(self):
        return self.changed_by

    @_history_user.setter
    def _history_user(self, value):
        self.changed_by = value

    def __str__(self):
        return self.name

# Approval Model (inherited from abstract model)
class ServerApproval(AbstractServer):
    action = models.CharField(
        max_length=10,
        choices=APPROVAL_ACTIONS,
        blank=False
    )
    approver = models.ForeignKey(User, blank=False, null=False, on_delete=models.CASCADE, related_name='server_approver')
    status = models.CharField(
        max_length=10,
        choices=APPROVAL_STATES,
        blank=False
    )
    datetime = models.DateTimeField(blank=False)
    comments = models.TextField(blank=True, null=True)

    def __str__(self):
        return self.name

Note the ‘abstract=True‘ flag in the abstract model’s meta class. This is to ensure that the abstract model is not created in DB and only the inherited models are. Perfect use-case for our requirement. You can read more about model inheritance from the Django official documentation. There are some fields specific to the primary model as it has django-simple-histiory enabled, while the approval model has fields such as action, status, approver very specific to the model to capture moderation details as the intention is to save these information as well.

Now that our model is ready, let’s see how we can use DRF to provide a form to the user to add/update changes. And these changes should be read from the primary model however saved in the approval model first. Now the same thing can be done using Django formset totally leaving out DRF. However as we already have DRF configured and things as such as permissions, field serialization already done, I found using DRF’s TemplateHTMLRenderer a much elegant solution.

Once DRF is setup, meaning you have necessary serializers, url routers in place, we would need an additional approval model view to render the form and also save the details in the approval model.

# Server edit form backend
class ServerEditForm(APIView):
    permission_classes = [IsAuthenticatedOrStaffUser,]
    renderer_classes = [TemplateHTMLRenderer]
    template_name = 'server_edit_form.html'
    style = {'vertical_style': {'template_pack': 'rest_framework/vertical'},
             'horizontal_style': {'template_pack': 'rest_framework/horizontal'}}
    
    def get(self, request, **kwargs):
        param = kwargs.get('param')
        # Pass users
        users = User.objects.exclude(id=request.user.id)
                
        # Render add server form
        if param == 'add':
            serializer = ServerApprovalWriteSerializer()
            return Response({'serializer': serializer, 'action': 'add', 'users': users, 'style': self.style})
        
        # Render update server form
        elif 'update' in param:
            server = get_object_or_404(Server, pk=kwargs.get('param').replace('update',''))
            serializer = ServerWriteSerializer(server)
            
            return Response({'serializer': serializer, 'server': server, 'users': users, 'style': self.style})

    def post(self, request, **kwargs):
        serializer = None
        param = kwargs.get('param')
        
         # Update status and requestor fields
        request.data._mutable = True
        request.data['status'] = 'Pending'
        request.data['approver'] = request.POST.get('approver')
        request.data['requestor'] = request.user.pk
        request.data['datetime'] = datetime.datetime.now()
            
        # Submit add server request
        if param == 'add':
            request.data['action'] = 'Add'
            name = request.data['name']
            ip = request.data['ip']
            
            # Validate if server exists in Server model
            if Server.objects.filter(name=name).exists() or Server.objects.filter(ip=ip).exists():
                return JsonResponse({'response': 'ERROR adding server
Server/IP already exists!'})
            
             # Validate if server exists in ServerApproval model
            if ServerApproval.objects.filter(name=name).exists() or ServerApproval.objects.filter(ip=ip).exists():
                server = ServerApproval.objects.get(Q(name=name) | Q(ip=ip))
                if (server.status == 'Pending' or server.status == 'On Hold'):
                    return JsonResponse({'response': 'ERROR adding server
Existing request found for server({0})/IP({1})! You may raise a new request once the existing request is Approved or Rejected'.format(name, ip)})
                else:
                    # Updating server approval entry
                    serializer = ServerApprovalWriteSerializer(server, data=request.data)
            else:
                # Add new entry
                serializer = ServerApprovalWriteSerializer(context={'request': request}, data=request.data)
        
        # Submit update server request
        else:
            try:
                request.data['action'] = request.POST.get('action').capitalize() 
                request.data['comments'] = request.POST.get('comments')
                
                existing_server = get_object_or_404(Server, pk=kwargs.get('param'))
                server = ServerApproval.objects.get(name=existing_server.name)
                serializer = ServerApprovalWriteSerializer(server, data=request.data)
            except ServerApproval.DoesNotExist:
                # Server does not exists in approval queue! Create new entry in approval queue
                serializer = ServerApprovalWriteSerializer(context={'request': request}, data=request.data)
    
        # Handle serializer error
        if not serializer.is_valid():
            return JsonResponse({'response': 'ERROR submitting server details
{0}'.format(str(serializer.errors))})
        
        # Save changes
        serializer.save()
        
        # Redirect to response page
        return JsonResponse({'response': 'Change successfully submitted to approval queue!'})

Now, the approval model APIView class as shown above, might look a little complicated however, much of it is because of the extra fields and requirement specific customizations I had. In simpler terms. what it does is, override the GET and POST methods of the APIView. The GET method returns the form fields which is rendered in template and shown to user to request for entry addition/updates. If you notice the serializers used in GET, for additions it uses ServerApproval serializer and for updations it uses Server serializer. This is done to get primary model fields for updations and approval model fields for new entries. For POST which actually performs the save(), only ServerApproval serializer is used, meaning changes will be read from primary model but will be saved into the new approval model only, which is what we want. Apart from this, there are some checks/validations to handle edge cases. (for e.g. User tries to update entry already requested and so on). There are some fields which are set explicitly, like requestor/datetime/status which basically is not rendered in the form as we do-not want the user to specify these.

Now, lets have a look at the form template and url patterns.

# urls.py

    url('^server_approval/?$', views.server_approval, name='server_approval'),
    url('^review_data/?$', manager.review_data, name='review_data'),
    url('^submit_review/?$', manager.submit_review, name='submit_review'),
    url('^server_edit_form/(?P[^/]+)', views.ServerEditForm.as_view(), name='server_edit_form'),

# server_edit_form.html

{% load static %}
{% load rest_framework %}
<html>
<body>
{% url 'server_edit_form' param='add' as add_url %}
{% url 'server_edit_form' param='update' as update_url %}

{% if request.get_full_path == add_url %}
	<br>
	<form name="addForm" action="proxy.php?url=https%3A%2F%2Fb2techblog.wordpress.com%2F%7B%25+url+%27server_edit_form%27+param%3D%27add%27+%25%7D" method="POST">
		{% csrf_token %}
		<table>
		<tr><td class="required">{% render_field serializer.component style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.name style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.ip style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.dc style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.environment style=style.vertical_style %}</td></tr>
		<tr><td>{% render_field serializer.type style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.state style=style.vertical_style %}</td></tr>
		<tr><td>{% render_field serializer.tags style=style.horizontal_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.group style=style.vertical_style %}</td></tr>
		<tr><td class="required">
		<label>Approver</label>
			<select name="approver" class="form-control">
			  {% for user in users %}
			  <option value="{{ user.pk }}" selected>{{ user }}</option>
			  {% endfor %}
			</select>
		</td></tr>
		<tr><td>{% render_field serializer.description style=style.horizontal_style %}</td></tr>
		<tr><td>{% render_field serializer.comments style=style.horizontal_style %}</td></tr>
		</table>
	</form><br>
	<center><font color=red><b>*</b></font><i> indicates <b>mandatory</b> fields</i></center>
{% elif update_url in request.get_full_path %}
	<br>
	<form name="editForm" action="proxy.php?url=https%3A%2F%2Fb2techblog.wordpress.com%2F%7B%25+url+%27server_edit_form%27+param%3Dserver.pk+%25%7D" method="POST">
		{% csrf_token %}
		<table>
		<tr><td class="required">
			<label>Action</label>
			<select name="action" class="form-control">
			  <option value="Update" selected>Update</option>
			  <option value="Delete">Delete</option>
			</select>
		</td>
		<tr><td class="required">{% render_field serializer.component style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.name style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.ip style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.dc style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.environment style=style.vertical_style %}</td></tr>
		<tr><td>{% render_field serializer.type style=style.vertical_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.state style=style.vertical_style %}</td></tr>
		<tr><td>{% render_field serializer.tags style=style.horizontal_style %}</td></tr>
		<tr><td class="required">{% render_field serializer.group style=style.vertical_style %}</td></tr>
		<tr><td class="required">
			<label>Approver</label>
			<select name="approver" class="form-control">
			  {% for user in users %}
			  <option value="{{ user.pk }}" selected>{{ user }}</option>
			  {% endfor %}
			</select>
		</td></tr>
		<tr><td>{% render_field serializer.description style=style.horizontal_style %}</td></tr>
		<tr><td>
			<label>Comments</label><br>
			<textarea name="comments" class="form-control"></textarea>
		</td></tr>
		</table>
	</form><br>
	<center><font color=red><b>*</b></font><i> indicates <b>mandatory</b> fields</i></center>
{% endif %}
</body>
</html>

I have removed css/js from the template to reduce the LOC, however you can see how the DRF form is rendered. Some fields for which I want custom data (e.g. User fields with only current user), I do-not use the form-renderer, however simple HTML inputs. If you go back to the APIView class, you will find, data for these custom fields are fed separately to the serializer. You can find more information about DRF form rendering in the DRF official documentation. One thing I would like to highlight is, to customize the UI (e.g. add validations/style etc) a lot of element overrides needs to be done as the out of the box styling for DRF forms is pretty basic.

So majority of the work is completed and the only thing pending is handling the approval model template/actions. I additionally enhanced the approval model template to display a comparison table for update requests which show original data vs changes done for the entry. Below in the views.py you may refer the function to achieve this.

# views.py

# Retrieve review details
@csrf_exempt
def review_data(request):
    review_data = []
    review_error = ""
    
    for item in request.POST.dict():
        entry = json.loads(item)
        existing_server = Server.objects.get(name=entry['host'])
        existing_tags = []
        for tag in existing_server.tags.all():
            existing_tags.append(tag.name)
            
        # Fetch existing server entry        
        try:
            review_data.append({'component': {'existing_component': existing_server.component.name, 
                                              'new_component': entry['component'], 
                                              'class': get_diff(existing_server.component.name, entry['component'])},
                                'host': {'existing_host': existing_server.name, 
                                         'new_host': entry['host'],
                                         'class': get_diff(existing_server.name, entry['host'])},
                                'ip': {'existing_ip': existing_server.ip, 
                                       'new_ip': entry['ip'],
                                       'class': get_diff(existing_server.ip, entry['ip'])},
                                'dc': {'existing_dc': existing_server.dc.name, 
                                       'new_dc': entry['dc'],
                                       'class': get_diff(existing_server.dc.name, entry['dc'])},
                                'env': {'existing_env': existing_server.environment.name, 
                                        'new_env': entry['env'],
                                        'class': get_diff(existing_server.environment.name, entry['env'])},
                                'type': {'existing_type': existing_server.type.name, 
                                         'new_type': entry['type'],
                                         'class': get_diff(existing_server.type.name, entry['type'])},
                                'state': {'existing_state': existing_server.state, 
                                          'new_state': entry['state'],
                                          'class': get_diff(existing_server.state, entry['state'])},
                                'tags': {'existing_tags': existing_tags, 
                                         'new_tags': entry['tags'],
                                         'class': get_diff(existing_tags, entry['tags'])},
                                'group': {'existing_group': existing_server.group.name, 
                                          'new_group': entry['group'],
                                          'class': get_diff(existing_server.group.name, entry['group'])},
                                'desc': {'existing_desc': existing_server.description, 
                                         'new_desc': entry['desc'],
                                         'class': get_diff(existing_server.description, entry['desc'])}
                              }) 
    
        except Exception as err:
            review_error = str(err)

    return  HttpResponse(json.dumps({"review_data": review_data, "review_error": review_error}, cls = DjangoJSONEncoder, indent = None, separators = (',', ':')),
                     content_type='application/json;charset=utf-8')

Basically, the approval model entry id is returned from the approval template and its corresponding primary model data is matched and returned back to the template to show the comparison.

For the final save, a simple model save action is triggered when the approval status is updated from the UI template.

# views.py

# Submit review
@csrf_exempt
def submit_review(request):
    review_data = []
    review_error = ""
    
    # Save entry based on action
    try:
        action = request.POST.get('action')
        server_approval = ServerApproval.objects.get(pk=request.POST.get('id'))
        
        if action == 'approved':
            server_approval.status = 'Approved'
            review_data.append({'response': 'Change APPROVED successfully!'})
        elif action == 'onhold':
            server_approval.status = 'On Hold'
            review_data.append({'response': 'Change status set to ON HOLD'})
        elif action == 'rejected':
            server_approval.status = 'Rejected'
            review_data.append({'response': 'Change REJECTED successfully!'})
        
        server_approval.datetime = datetime.datetime.now()
        server_approval.save()
    
    except Exception as err:
            review_error = str(err)
    
    return  HttpResponse(json.dumps({"review_data": review_data, "review_error": review_error}, cls = DjangoJSONEncoder, indent = None, separators = (',', ':')),
                     content_type='application/json;charset=utf-8')

All invocations is done using AJAX and the response is returned as in the functions above to display to the user.

When a model approval change is saved with status approved, a post-save signal is used to save the entry to the primary model as below:

# Signal to handle server approval
@receiver(post_save, sender=ServerApproval)
def create_update_approved_server(sender, instance, **kwargs):
    # Add server entry on approval
    if instance.status == 'Approved' and instance.action == 'Add':
        server_obj = Server.objects.create(name = instance.name,
                                  component = instance.component,
                                  ip = instance.ip,
                                  dc = instance.dc,
                                  environment = instance.environment,
                                  type = instance.type,
                                  state = instance.state,
                                  group = instance.group,
                                  description = instance.description,
                                  requestor = instance.requestor,
                                  changed_by = instance.approver
                                  )
        server_obj.tags.add(*instance.tags.all())
        
    # Update server entry on approval
    if instance.status == 'Approved' and instance.action == 'Update':
        server_obj, created =  Server.objects.update_or_create(name = instance.name,
                                                               defaults = {'component': instance.component,
                                                                         'ip': instance.ip,
                                                                           'dc': instance.dc,
                                                                           'environment': instance.environment,
                                                                           'type': instance.type,
                                                                           'state': instance.state,
                                                                           'group': instance.group,
                                                                           'description': instance.description,
                                                                           'requestor': instance.requestor,
                                                                           'changed_by': instance.approver
                                                                           }
                                                               )
        server_obj.tags.add(*instance.tags.all())
    
    # Delete server entry on approval
    if instance.status == 'Approved' and instance.action == 'Delete':
        server_obj  = Server.objects.get(name = instance.name)
        server_obj.delete()

As I had a many-to-many field, had to add those after the model was saved using the .add () method as shown above. The signal ensures to add an entry to the primary table if not present (for add requests) and update an existing one if already present (update requests).

I am using datatables with jquery for the template. Snippet to display the approval form (in jquery dialog)

$("#actionDialog").dialog({
	  	title: dialogTitle,
	  	position : [ 'center' ],
      	width: '500px',
      	height: 'auto',
      	autoOpen: false,
      	closeOnEscape: false,
      	modal: true,
      	open: function () {
    		$("#actionStatus").load(url, function(data, status, jqXGR) {
    			if(status == 'error') {
    				$("#actionWait").html('<br><font color=red><b>Error retrieving server details!</b></font>');
    			} else {
    				$("#actionWait").remove();
    			}
        	});
      	},
      	buttons:[
      	   {
			  text: 'Save', 
			  id: 'actionBtn',
			  click: function () {
				if (validateForm()) {
					$("#actionDialog").scrollTop("0");
					$("#actionDialog").prepend("<center id='actionWait'><br><div class='spinner'><br><br></div></center>");
					$("#actionBtn").addClass("ui-state-disabled").attr("disabled", true);
					
					// AJAX call to submit changes
					$.ajax({
						type:'POST', 
				    	url: $("form[name='" + formName + "']").attr('action'),
				    	data: $("form[name='" + formName + "']").serialize(),
				    	success: function(message) {
				    		$("#actionStatus").html('<center><br><br>' + message.response + '</center>');
				    	},
				  		error: function(error) {
		              		$("#actionStatus").html('<center><font color=red><br><b>ERROR submitting changes!</b><br>' + error + '</font></center>');
		            	},
		       	     	complete: function(jqXHR, textStatus) {
		       	     		$("#actionWait").remove();
		    	     	}
				  	});
				}
			  }
			},
			{
		  	  text: 'Close', 
		  	  click: function () { 
			  	resetActionDialog();
		   		$(this).dialog('close');
		  	  }
		   }
	  	]
	}).dialog('open');

I hope you have got a basic understanding of how we can achieve moderation in Django models. The solution works seamlessly and also gives us control to customize stuff based on requirements.

I think thats a pretty long post. Hope you don’t get bored reading it 😉

Of late, I was creating a dashboard in Django for infra inventory tracking and was looking how best to do it. Basically, the requirements were:

1. Tabular view of servers and related details
2. Audit and History views for any change done
3. RESTful services to expose API’s to get/post/patch/delete data
4. Fast performance with minimal lag
5. Scalable/Configurable

I have worked on such requirements before, however the API part was something new. Enter the Django Rest Framework (DRF). Am listing down some of the challenges I faced while developing this.

To start with below are the modules I used.

1. Django 3.0.8
2. Django Filters
3. Django Simple History
4. Django YASG for Swagger/Redoc
5. Datatables (UI)
6. Memcached (Python Memcached, Django Memcache Status, Django Clearcache)
7. Postgres (DB)
8. Django Auth LDAP (AuthN/AuthZ)

The base was pretty simple, and it was a not-so-complex model based view that I wrote. Basically, a server model with server host, ip and other details. It had some foreign key’s like DC, Environment etc.

Now, with DRF, serializing the fields the way I want was the first hurdle. I wanted different serializers for read and write modes, where read required no authentication, better display of API data (especially the foreign keys) and eager loading enabled. Ended up with a mixin to achieve this. Note the SELECT FIELDS for 1-1 mapped fields and PREFETCH_FIELDS for 1-Many in the eager loading mixin. This was done to get around the infamous n+1 queries issue.

#serializer.py

# Eager load mxin
class EagerLoadingMixin:
    @classmethod
    def eager_loading(cls, queryset):
        if hasattr(cls, "_SELECT_FIELDS"):
            queryset = queryset.select_related(*cls._SELECT_FIELDS)
        if hasattr(cls, "_PREFETCH_FIELDS"):
            queryset = queryset.prefetch_related(*cls._PREFETCH_FIELDS)
        return queryset

class ServerReadSerializer(serializers.ModelSerializer, EagerLoadingMixin):
    component = ComponentSerializer(read_only=True)
    dc = DcSerializer(read_only=True)
    environment = EnvironmentSerializer(read_only=True)
    type = TypeSerializer(read_only=True)
    tags = TagSerializer(read_only=True, many=True)
    group = GroupSerializer(read_only=True)
    changed_by = UserSerializer(read_only=True)
    
    _SELECT_FIELDS = ['component','dc', 'environment', 'type', 'group', 'changed_by']
    _PREFETCH_FIELDS = ['tags',]
    
    class Meta:
        model = Server
        fields = '__all__'

class ServerWriteSerializer(serializers.ModelSerializer):   
    class Meta:
        model = Server
        fields = '__all__'
        read_only_fields = ('changed_by',)
         
    def create(self, validated_data):
        user = self.context['request'].user
        validated_data['changed_by'] = user
        return super().create(validated_data)

Also note, the ‘changed_by‘ readonly field, which was added to audit changes done by user, using the simple-history module. Will come to that in a bit, however the readonly setting for this field was important as we do-not want users to select this field while making a post request to the API. Also see the create method override, which is done to set the changed_by field to the current logged in user.

For permissions, had a custom permission class and the noteworthy django-filter which I find so much better than the default search field provided by DRF. Check the views.py file below:

#views.py

# Serializer mixins    
class ReadWriteSerializerMixin(object):
    read_serializer_class = None
    write_serializer_class = None

    def get_serializer_class(self):        
        if self.action in ["create", "update", "partial_update", "destroy"]:
            return self.get_write_serializer_class()
        return self.get_read_serializer_class()

    def get_read_serializer_class(self):
        assert self.read_serializer_class is not None, (
            "'%s' should either include a `read_serializer_class` attribute,"
            "or override the `get_read_serializer_class()` method."
            % self.__class__.__name__
        )
        return self.read_serializer_class

    def get_write_serializer_class(self):
        assert self.write_serializer_class is not None, (
            "'%s' should either include a `write_serializer_class` attribute,"
            "or override the `get_write_serializer_class()` method."
            % self.__class__.__name__
        )
        return self.write_serializer_class

# Custom permission
class IsAuthenticatedOrSuperUser(BasePermission):
    def has_permission(self, request, view):
        return bool(
            request.method in  ('GET', 'HEAD', 'OPTIONS') or
            request.user and
            request.user.is_authenticated and 
            request.user.is_staff and
            request.user.is_superuser
        )
        
# Viewsets
class ServerViewSet(ReadWriteSerializerMixin, viewsets.ModelViewSet):
    queryset = Server.objects.all()
    queryset = ServerReadSerializer.eager_loading(queryset)  # Eager loading to improve performance
    read_serializer_class = ServerReadSerializer
    write_serializer_class = ServerWriteSerializer
    permission_classes = [IsAuthenticatedOrSuperUser,]
    filter_backends = (filters.DjangoFilterBackend, )
    filterset_fields = {'component__name': ['exact', 'iexact'], 
                        'name': ['exact', 'iexact'], 
                        'ip': ['exact', 'iregex'], 
                        'dc__name': ['exact', 'iexact', 'iregex'], 
                        'environment__name': ['exact', 'iexact'], 
                        'type__name': ['exact', 'iexact'], 
                        'state': ['exact', 'iexact'], 
                        'tags__name': ['exact', 'iexact', 'icontains', 'iregex'], 
                        'group__name': ['exact', 'iexact'], 
                        'description': ['exact', 'icontains', 'iregex']
                        }

This worked perfect, and with the eager loading setup, could see some performance improvement as well. But it was not enough.

Enter memcached. There can be a debate between memcached vs redis. However I just wanted a simple caching backend which just does caching. Redis is so much more to be frank and thus I decided to carry on with memcached. Did not go with the default memcached middleware which caches basically all sites. Had a custom caching configured with a preset key, so that the invalidation would be easier. Ignore, the ‘url filtering‘ logic below as that’s something my requirement specific, but otherwise the caching is quite straight forward.

# Handle caching
def cache_response(view, url, cache_name):
    # Do-not serve from cache if filter is passed
    if '&' in url:
        status, response = send_api_request(url, view, None, None)
        if status != 200:
            raise Exception('Error fetching data from API: ' + response.content)
        
    else:
        # Get response from cache
        response = cache.get(cache_name)
        
        # Invoke POST call to get data from DRF if cache is not set
        if not response:   
            status, response = send_api_request(url, view, None, None)
            
            if status != 200:
                raise Exception('Error fetching data from API: ' + response.content)
            else:
                # Set cache
                cache.set(cache_name, response, None)  
    
    return response

For invalidation, utilized the simple-history post-save signal, which I also used, to add audit entries whenever a model was updated.

@receiver(post_create_historical_record)
def post_create_record_callback(sender, **kwargs):
    # Define audit action
    # Get model class name
    model_name = kwargs.get('instance').__class__.__name__
    action = 'Unknown'
    history = kwargs.get('history_instance')
    history_type = history.history_type
    
    if history_type == '+':
        action = 'Added'
    elif history_type == '-':
        action = 'Deleted'
    elif history_type == '~':
        action = 'Updated'
    
    # Add audit entry
    audit = Audit(name = action, 
                  logs = '{0} {1}'.format(model_name, history.name), 
                  datetime = datetime.datetime.now(), 
                  user = kwargs.get('history_user'))
    audit.save()
    
    # Clear cache
    if cache.has_key('audit_cache'):
        cache.delete('audit_cache')
        
    if model_name == 'Server' and cache.has_key('server_cache'):
        cache.delete('server_cache')

One thing to highlight here is the ‘.__class__.__name__’ function. This function returns the class name of an object, thus allowing to get the model name modified and in turn used this information to invalidate the appropriate cache key. Would also like to point, that there is no direct way to clear a prefix cache which I tried before this approach. With this approach I have more control on what to cache and when to invalidate selective cache keys. This signal is very specific to django-simple-history module, so google up the same if some of the kwargs options are not clear. Practically the cache set had a None timeout, meaning it would never get invalidated on its own and would only get invalidated with any change with the model which is exactly what I wanted. This led to an almost 10X performance improvement. From ~ 20 seconds the response time came down to ~ 2 seconds.

While integrating simple-history found this very useful to have the history enabled in admin site with my custom model fields listed.

#admin.py

@admin.register(Server) 
class ServerAdmin(SimpleHistoryAdmin):
    model = Server
    list_display = ['component', 'name', 'ip', 'dc', 'environment', 'type', 'state', 'group']
    search_fields = ['component__name', 'name', 'ip', 'dc__name', 'environment__name', 'type__name', 'state', 'group__name']
    filter_horizontal = ('tags',)
    exclude = ['changed_by',]
    
    def save_model(self, request, obj, form, change):
        obj.changed_by = request.user
        super().save_model(request, obj, form, change)Wi

The save_model override is again for the changed_by field. The field is in the exclude list as well as we don’t want to show it to anyone even in admin site.

The model reference:

#models.py

class Server(models.Model):
    component = models.ForeignKey(Component, blank=False, null=False, on_delete=models.CASCADE)
    name = models.CharField(max_length=50, unique=True, blank=False, null=False)
    ip = models.GenericIPAddressField(unique=True, blank=False, null=False)
    dc = models.ForeignKey(Dc, blank=False, null=False, on_delete=models.CASCADE)
    environment = models.ForeignKey(Environment, blank=False, null=False, on_delete=models.CASCADE)
    type = models.ForeignKey(Type, blank=True, null=True, on_delete=models.CASCADE)
    state = models.CharField(
        max_length=10,
        choices=STATES,
        blank=False
    )
    tags = models.ManyToManyField(Tag, blank=True, related_name='tags')
    group = models.ForeignKey(Group, blank=False, null=False, on_delete=models.CASCADE)
    description = models.TextField(blank=True, null=True)
    changed_by = models.ForeignKey(User, blank=False, null=False, on_delete=models.CASCADE)
    history = HistoricalRecords()

    @property
    def _history_user(self):
        return self.changed_by

    @_history_user.setter
    def _history_user(self, value):
        self.changed_by = value

    def __str__(self):
        return self.name

You can see the simple-history settings configured and changed_by field introduced.

Hope that clears thing up. Let’s get to the template now. I used datatables for the presentation layer, and a djangorestframework-datatables module to fetch the data from the API itself which my app would be exposing, rather than querying the models again. While this may look like a round about way of achieving things, but would definitely remove a lot of lines of code and also enable filtering capabilities which django-filter provides for DRF. Finally this would also kind of be a unit testing for the API’s we write. The approach is definitely arguable, but it worked for me and worked really good.

So just had to use the DRF APIRequestFactory to send a GET request to my API with an added ‘?format=datatables‘ param and wohoo, the response was well-formatted and ready for datatables to consume.

The request function using APIRequestFactory is as below:

# Send DRF post request
def send_api_request(url, view, data, user):
    request = None
    factory = APIRequestFactory()
    if user:
        request = factory.post(url, data, format='json')
        force_authenticate(request, user=user)
    else:
        request = factory.get(url)
    
    response = view(request)
    response.render()
    
    return response.status_code, response.content

However, one challenge I faced was to construct the URL. I did not want to use the nasty getpath/host commands and hardcode the context. This is what I found after much detailed lookup of the DRF documentation:

url = reverse('reinvent_rest_api:server_api-list')

where in urls.py:

#urls.py

router.register('server', views.ServerViewSet, basename='server_api')
path('api/v1/', include((router.urls, 'reinvent'), namespace='reinvent_rest_api')),

The router configurations for ViewSet is well documented in DRF site, so not going into that detail. But the above worked perfect. Note the ‘-list ‘ with the basename. This is something that DRF sets. You can refer the DRF docs to know more.

So we are pretty much close to the end. Some more enhancements I did was on the UI side (using JQuery/Colvis/Buttons/Menu.js/Intro.js) just to make the UI more appealing.

On the memcached management front, installed 2 modules, django-memcache-status, django-clearcache, which gave me the options to see the memcached stats and also clear the cache from Admin site.

That’s pretty much about it. The end result is something that I am really liking ! 😉