When I first started developing an Angular application everything seemed to go well. Comparing to React, a lot of things were implemented out of the box. Until I reached the time when I needed to make some components of my application dynamic.

The problem

Almost every admin panel SPA looks the same. We have a sidebar for navigation, a topbar and the main content, that represents a work area of the application. This layout is UX-friendly, however sometimes we need to change not only the main layout, but other parts like a topbar or a sidebar.

In React this can be implemented easily with a few lines via <Router> and <Route> (in Angular too, as we will see later). All suggested solutions, that I googled, were too complex, and required some tricks. They were not hard to understand, but required steps that looked overcomplicated.

Possible solutions:

• Subscribe to route events and change components depending on the current route
• Via Component Factory Resolver and Component Refs
• ViewContainerRef
• … etc

Sample project

We need some code to play with. There is a great admin panel https://github.com/akveo/ngx-admin. It has a nice layout and design. I have forked this repo for the tutorial. Clone my repo using the following command

I won’t cover the structure of this project, hence this is out of scope of the tutorial. We will go step by step, hope everything will be clear enough. Navigate to the cloned directory and execute the following command to install depenendencies.

To run development server execute the next command.

After the application is compiled you should be able to access it via http://localhost:4200/

Agenda

The main idea is to add some specific components/elements to the topbar and the sidebar for specific pages.
Let’s take two pages for this tutorial.

1. Table Page. We will add two buttons in the Topbar for adding and deleting users.
2. IOT-Dashboard Page. Here we will add one spinner in the Topbar and action buttons into the Sidebar.

Furthermore we will change this static title to a dynamic one using Routes Data.

Routes extra data

Let’s start from an easier task. We will make our title to changin according to a selected currenlty selected page.
To pass extra data to routes define a property named data like that.

Next open the Header Component file (header.component.ts).
Inject ActivatedRoute and Router as constructor dependencies as follows:

Now we need to subscribe to route events to get extra data.

Of course you can adjust this subscription for your needs, but it should be clear what is going here. Call this method inside the ngOnInit hook. Finally add the pageTitle field and use it in the template.

As a result you should get title chainging while navigating between pages of the application.

This was just a bonus, let’s move to the main part.

Page specific Components in Tobpar and Sidebar

First of all, we need to define a new router outlet in places where we need our dynamic components.
Open the header.component.html file and add new outlet.

…

Also add RouterModule import to the ThemeModule. This step is required to recognize the router-outlet tag.

const BASE_MODULES = [CommonModule, RouterModule, FormsModule, ReactiveFormsModule];

Secondly, add another outlet to the Sample Layout Component (sample.layout.ts)

…

…

Finally, in order to add page specific content to the created outlets, firstly go to the Table Routing module (tables-routing.module.ts) and change it to look like that.

const routes: Routes = [{ path: ”, children: [{ path: ‘smart-table’, children: [ { path: ”, data: {title: ‘Tables’}, component: SmartTableComponent, }, { outlet: ‘header-top’, path: ”, component: TablesHeaderComponent, }], }], }];

As agreed, we need to add two buttons to the header on the Tables page. Define a simple dummy component.

@Component({ selector: ‘ngx-table-header-component’, template: ` ` , }) export class TablesHeaderComponent { }

After that, wait for compilation and navigate to the route http://localhost:4200/#/pages/tables/smart-table
In case everything is done right you should get the following output.

As you could see, we have just added desired behavior without refs and other suggested techiques.
To complete our task let’s move our attention to the IOT Dashboard page. Routing configuration for that page is located in the root pages routing module (pages-routing.module.ts).
As you may already guessed it should be defined in the following way.

{ path: ‘iot-dashboard’, children: [ { data: {title: ‘IOT Dashboard’}, path: ”, component: DashboardComponent, }, { outlet: ‘header-top’, path: ”, component: DashboardHeaderComponent, }, { outlet: ‘sidebar-top’, path: ”, component: DashboardSidebarComponent, }, ], }

Create components, declare them in module and see the result.
For the header part I will use the following layout:

@Component({ selector: ‘ngx-dashboard-header-component’, template: ` Some card content. ` , }) export class DashboardHeaderComponent { }

The sidebar component is implemented in this fashion:

@Component({ selector: ‘ngx-dashboard-sidebar-component’, template: ` ` , }) export class DashboardSidebarComponent { }

Declare components in the module, recompile an application and you should see a similar result.

Everything work pretty well.

Conclusion

In this tutorial I’ve described a declarative way for changing components dynamically based on the current route without any references. You can use this method for other cases in your application. Hope this article will be useful for someone.
You can find full project source code at this repository:

Source code

The post Page Specific Dynamic Angular Components using Child Routes appeared first on Oleksandr Molochko.

Dagger 2 is a compile-time DI framework that is becoming increasingly popular in Android development. Using Dagger 2 can make application structure more scalable and maintainable. However, in this post I am not going to tell you about all benefits of using Dagger 2. This article is intended for developers that already had some experience using this framework and are eager to learn it a little bit deeper. One feature of Dagger 2 is the concept of Scope. Scopes are not easy to understand from the first time of use and require some effort to get used to them for applying them correctly in a application.

Introduction

If you are reading this article, I assume that you already have got your hands dirty playing with Dagger 2 and familiar with Component, Provider, Inject annotations and their purpose.

Before diving into the Scope mechanism, you should have a quite decent understanding of when and why the dependency injection mechanism is applied. Furthermore, I won’t describe some core theoretical concepts like a dependency graph, components and providers.

All code from the article can be found on GitHub.

The Definition of The Scope

What is a scope?

The definition above may be rough and not totally correct from some technical aspects, but I want to stress one important thing, that all objects in a scope will live as long as a Dagger component instance lives. All in all, there is nothing special about components or scopes, all usual Java/Android garbage collecting and reference counting mechanisms work here in the same way. The same instance of a scoped dependency is injected, when the same component is used.

Don’t misunderstand me, if your component is destroyed but a dependency instance is still being kept by some object, of course it won’t be destroyed, but next time you inject dependencies another instance will be provided.

Scopes are basically used during compile time for generating injectors and providers. We will see in a while how generated classes are used in runtime to maintain scoped dependencies.

A sample Project

Let’s create a really simple project for understanding the matter. I will take the liberty of violating all design best practices and principles for the sake of simplicity. (the best excuse for writing terrible code ). I have used some boilerplate classes like BaseActivity, BaseFragment, explanation of these classes is out of scope of this article, hence you can find full code on Github.

Firstly, we need to define some scopes to have something to investigate. I have defined quite common scopes that are usually used in Android Development.

As you can see from the picture above all scopes are really tied to Android SDK classes and have respective names. Why do we usually have such scopes in android applications? Because we as developers are driven by frameworks, in that case it is Android SDK and we have not much control over the lifecycle of core system components like activities, fragments.

• Application scope – this is the most long living scope, that exists during the whole application lifetime. Components and modules within that scope usually contain Singleton/Global dependencies, like Repositories, API Services, etc.
• Activity scope – this scope, as the name suggests, lives as long as an activity does. Usually, I use this scope for a self-contained modules, a single module implements a specific case of an application. As far as the Android Activity is just a detail, it acts as a container and a router. Scoped components provide dependencies for a specific use case of an application.
• Fragment scope – the most short living scope in our case, as it is kept by a fragment. Dependencies in that scope are usually related to the presentation and UI interaction logic.

The app itself consists of two activities and four fragments. Each of activities is responsible for keeping two fragments respectively. You can switch between fragments inside a single activity or switch to another activity.

The DI component diagram represents general project structure.

Please note the color of each component, it is assigned according to the scope of the component. As we have multiple components and they form the hierarchy, we need to define relationships between components. I have used the @Subcomponent relationship between child and parent components. Subcomponents have an access to the entire objects graph from parent components, including transitive dependencies. We will see later what happens under the hood with parent and child components.

Names are chosen just to make it clear which Android class a component is related to. I would have never used such names in real application.

Custom scope annotations

First of all, we need to define custom scopes, from the code perspective, a scope is nothing more than annotation. We will have multiple Scope annotations:

• @PerApplication – the global scope used for the Application component.
• @PerActivity – this scope is used mainly for base activities. Dependencies that are provided in that scope are usually from Android SDK, like Resources, Layout Inflater, Context. No activity specific dependencies are provided.
• @PerScreen – this scope has the same lifetime as the previous one. All scoped components are available during activity lifecycle. However, components in that scopes provide dependencies that are specific for the Activity/Module.
• @PerFragment – the narrowest scope that exists only during the Fragment lifetime. Components within the scope inject dependencies required by a single view. (The Fragment is just a view).

A custom scope is defined as follows:

In order to create a Subcomponent, we need to access a parent component, hence let’s define an interface for that purpose.

{ C getComponent(); }

It can be used as follows. Casting is not the best idea, however it is almost impossible to live without this feature in the Android world.

private void initDependencyComponents() { ApplicationComponent applicationComponent = ((ProvidesComponent) getApplication()).getComponent(); mActivityComponent = applicationComponent.plusActivityComponent(new ActivityModule(this)); mActivityComponent.inject(this); }

Unscoped Dependencies

First of all we need to understand how do Unscoped dependencies behave and implemented in generated code.

Let’s create the Application Component and related modules.

@PerApplication @Component(modules = { ApplicationModule.class}) public interface ApplicationComponent { // Injection methods void inject(Dagger2ScopeInternalsApplication mainApplication); }

Also create the Application module with some dependencies.

@Module public class ApplicationUnscopedModule { Application mApplication; public ApplicationUnscopedModule(Application application) { mApplication = application; } @Provides Application provideApplication() { return mApplication; } @Provides @Named(GLOBAL_APP_CONTEXT) public Context provideApplicationContext() { return mApplication.getApplicationContext(); } @Provides public CarDataRepositoryContract provideCarDataRepository() { return new CarDataRepository(); } }

I have defined a simple interface CarDataRepositoryContract, without any method and implemented it. Next we need to set dependencies in Dagger2ScopeInternalsApplication, BaseActivity and MainActivity classes. Our main application class should look something like that for now.

public class Dagger2ScopeInternalsApplication extends Application implements ProvidesComponent { @Inject CarDataRepositoryContract mCarRepo; private ApplicationComponent mApplicationComponent; @Override public void onCreate() { super.onCreate(); buildApplicationComponent(); mApplicationComponent.inject(this); } private void buildApplicationComponent() { mApplicationComponent = DaggerApplicationComponent.builder() .applicationUnscopedModule(new ApplicationUnscopedModule(this)) .build(); } @Override public ApplicationComponent getComponent() { return mApplicationComponent; } }

As you can see from the code above, we are building the Application Component, we need to store it as the class field to be able to access it from activities.

In activities to get the Application Component we are casting the Application class to the ProvidesComponent interface. As result in onCreate method we are injecting dependencies in the following way.

public abstract class BaseActivity extends AppCompatActivity implements ActivityController { @Inject CarDataRepositoryContract mCarRepo; //… Other variables protected ApplicationComponent mAppComponent; @Override protected void onCreate(@Nullable Bundle savedInstanceState) { super.onCreate(savedInstanceState); initDependencyComponents(); //… Further initialization } private void initDependencyComponents() { mAppComponent = ((ProvidesComponent) getApplication()).getComponent(); mAppComponent.inject(this); } }

Finally, in the MainActivity class we also injecting dependencies (only one dependency) using the component from BaseActivity as follows.

public class MainActivity extends BaseSingleFragmentActivity implements MainFirstFragment.SecondFragmentRouter { @Inject CarDataRepositoryContract mCarRepoOther; //… Other variables @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); injectDependencies(); //… Further initialization } private void injectDependencies() { mAppComponent.inject(this); } }

You may wonder why do we need CarDataRepositoryContract in BaseActivity and MainActivity. In real world this doesn’t have any sense, but for our tutorial it will show how unscoped dependencies are provided. Let’s figure this out.

Build and launch the application, but set brakepoints on lines in all classes that have CarDataRepositoryContract as a dependency (e.g. the onCreate method).

CarDataRepository@3611 CarDataRepository@3778 CarDataRepository@3779

As you can see from results above, we got a new instance every time we requested dependencies. Even if you call the inject method multiple times in the same class you will get a new instance anyway.

Definitely, if we injected the Application dependency in activities it would be set to the same instance, as long as this object itself is a singleton.

You can get the code for that example from this commit.

To conclude:

• If you defined a provided method without the Scope annotation, anytime you request the dependency you will get a new instance.
• As you may have noted the Application Component was annotated with the @PerApplication annotation, but that didn’t change the situation.
• Calling injection from the same component (with unscoped providers) in the same class several times will result in new instances of dependencies.

Internals of generated classes

It’s time to look under the hood. Open the implementation of the ApplicationComponent interface.

Locating generated classes

You can find all generated files in the generatedJava directory or by navigating to implementations of the interface. Another way is just to find the class with the Dagger prefix for your component, for instance ApplicationComponent will have DaggerApplicationComponent implementation generated.

Here is the full code of the ApplicationComponent implementation, if you followed the described steps you should have a similar one.

public final class DaggerApplicationComponent implements ApplicationComponent { private Provider provideCarDataRepositoryProvider; private MembersInjector dagger2ScopeInternalsApplicationMembersInjector; private MembersInjector mainActivityMembersInjector; private MembersInjector baseActivityMembersInjector; private DaggerApplicationComponent(Builder builder) { assert builder != null; initialize(builder); } public static Builder builder() { return new Builder(); } @SuppressWarnings(“unchecked”) private void initialize(final Builder builder) { this.provideCarDataRepositoryProvider = ApplicationUnscopedModule_ProvideCarDataRepositoryFactory.create( builder.applicationUnscopedModule); this.dagger2ScopeInternalsApplicationMembersInjector = Dagger2ScopeInternalsApplication_MembersInjector.create(provideCarDataRepositoryProvider); this.mainActivityMembersInjector = MainActivity_MembersInjector.create(provideCarDataRepositoryProvider); this.baseActivityMembersInjector = BaseActivity_MembersInjector.create(provideCarDataRepositoryProvider); } @Override public void inject(Dagger2ScopeInternalsApplication mainApplication) { dagger2ScopeInternalsApplicationMembersInjector.injectMembers(mainApplication); } @Override public void inject(MainActivity mainActivity) { mainActivityMembersInjector.injectMembers(mainActivity); } @Override public void inject(BaseActivity baseActivity) { baseActivityMembersInjector.injectMembers(baseActivity); } public static final class Builder { private ApplicationUnscopedModule applicationUnscopedModule; private Builder() {} public ApplicationComponent build() { if (applicationUnscopedModule == null) { throw new IllegalStateException( ApplicationUnscopedModule.class.getCanonicalName() + ” must be set”); } return new DaggerApplicationComponent(this); } public Builder applicationUnscopedModule(ApplicationUnscopedModule applicationUnscopedModule) { this.applicationUnscopedModule = Preconditions.checkNotNull(applicationUnscopedModule); return this; } } }

Components

Let’s start our journey from Components. Components are major building blocks of the Dagger 2 library. A Component implementation acts as a container (i.e. orchestrator) for all other players of Dagger 2. When you build a component you have to provide all dependencies that cannot be resolved automatically. Anyway, we are somehow addicted to Android SDK, hence Context, Application, Service classes are usually provided to modules to construct corresponding components.

Let’s discuss some snippets of the generated implementation. As you could see the creation of the component itself happens inside the Builder nested class. As a result you won’t be able to create the Component directly or to create it without provided required modules.

Components are not injecting dependencies by themselves, they just delegate this task to the MembersInjector classes.

@Override public void inject(Dagger2ScopeInternalsApplication mainApplication) { dagger2ScopeInternalsApplicationMembersInjector.injectMembers(mainApplication); }

The creation of MembersInjectors happens after all Providers are created in the initialize() method.

MembersInjectors

MembersInjector classes are exactly responsible for injecting dependencies into dependent objects. Roughly speaking any class that requires dependencies and has at least one @Inject annotation has it’s corresponding injector.

Let’s have a look at the MainActivity_MembersInjector class.

public final class MainActivity_MembersInjector implements MembersInjector { private final Provider mCarRepoAndMCarRepoOtherProvider; public MainActivity_MembersInjector( Provider mCarRepoAndMCarRepoOtherProvider) { assert mCarRepoAndMCarRepoOtherProvider != null; this.mCarRepoAndMCarRepoOtherProvider = mCarRepoAndMCarRepoOtherProvider; } public static MembersInjector create( Provider mCarRepoAndMCarRepoOtherProvider) { return new MainActivity_MembersInjector(mCarRepoAndMCarRepoOtherProvider); } @Override public void injectMembers(MainActivity instance) { if (instance == null) { throw new NullPointerException(“Cannot inject members into a null reference”); } net.crosp.android.dagger2scopeinternals.base.ui.activity.BaseActivity_MembersInjector .injectMCarRepo(instance, mCarRepoAndMCarRepoOtherProvider); instance.mCarRepoOther = mCarRepoAndMCarRepoOtherProvider.get(); } public static void injectMCarRepoOther( MainActivity instance, Provider mCarRepoOtherProvider) { instance.mCarRepoOther = mCarRepoOtherProvider.get(); } }

In order to a MembersInjector be created all dependencies should be provided via Provider classes passed to the create method.

Please pay attention to the highlighted lines (20,21,21). I have intentionally not mentioned this while describing the application structure. When a parent class has dependencies they should be satisfied before injecting dependencies in the child class. And that totally makes sense, otherwise you could get NullPointer Exceptions thrown. Dagger 2 is smart enough to detect such kind of relationships while building a dependency graph and generating classes. You can even remove the void inject(BaseActivity baseActivity) method, but the BaseActivity_MembersInjector class will still be generated.

When actual injection happens a MembersInjector instance delegates this request to a corresponding Provider.

Providers

Now, we have reached the most interesting part from that article’s perspective – Provider. Provider is a guy that uses created modules to provide dependencies. Let’s have a look at the implementation of the ApplicationUnscopedModule_ProvideCarDataRepositoryFactory.

public final class ApplicationUnscopedModule_ProvideCarDataRepositoryFactory implements Factory { private final ApplicationUnscopedModule module; public ApplicationUnscopedModule_ProvideCarDataRepositoryFactory( ApplicationUnscopedModule module) { assert module != null; this.module = module; } @Override public CarDataRepositoryContract get() { return Preconditions.checkNotNull( module.provideCarDataRepository(), “Cannot return null from a non-@Nullable @Provides method”); } public static Factory create(ApplicationUnscopedModule module) { return new ApplicationUnscopedModule_ProvideCarDataRepositoryFactory(module); } }

At line 14 you can see a direct call to the ApplicationUnscopedModule module. That’s it, we finally reached the source of the dependency.

I hope you have better understanding of how injection happens and flows. It’s time to understand how scopes affect all this stuff.

Scoped Dependencies

We need to create other components as they were defined at the diagram in the beginning of the article. For the simplicity I will have a single module per a component.

Due to the fact that child components and parent components are connected via the @Subcomponent mechanism, to create a child component we need to have access to the parent component. A parent component is responsible for creating a sibling component, but all required dependencies to modules are provided by the child component.

Subcomponents

Let’s define subcomponents according to the diagram and modify existing ApplicationComponent. I think it will be better to start from the bottom and move upwards.

MainFirstViewComponent

@PerFragment @Subcomponent(modules = {MainFirstFragmentModule.class}) public interface MainFirstViewComponent { void inject(MainFirstFragment fragment); }

This component corresponds to the MainFirstFragment class and we need only one dependency to be injected in it, that is a router that will handle navigation between views.

public class MainFirstFragment extends BaseFragment { // Callbacks @Inject SecondFragmentRouter mRouter; // … other stuff public interface SecondFragmentRouter { void onSwitchToSecondFragment(); } }

The containing activity will acts as a router, therefore we provide this dependency in the following way.

@Module public class MainFirstFragmentModule { @Provides @PerFragment public MainFirstFragment.SecondFragmentRouter provideSecondFragmentRouter(Activity containerActivity) { return (MainActivity) containerActivity; } }

MainScreenModule

@PerScreen @Subcomponent(modules = {MainScreenModule.class}) public interface MainScreenComponent { void inject(MainActivity activity); MainFirstViewComponent plusMainFirstViewComponent(); MainSecondViewComponent plusMainSecondViewComponent(); }

As far as we are using @Subcomponents we need to provide creator methods in the parent component to create children (plus methods). The Activity acts as a router and injected into both of child fragment components, albeit providing each component only required methods (Interface Segregation).

public class MainActivity extends BaseSingleFragmentActivity implements ProvidesComponent, MainFirstFragment.SecondFragmentRouter, MainSecondFragment.FirstFragmentRouter { // … @Override public void onSwitchToSecondFragment() { replaceFragment(MainSecondFragment.class, false); } @Override public void onSwitchBackToFirstFragment() { replaceFragment(MainFirstFragment.class, false); } @Override public void switchToSecondaryActivity() { startNewActivity(SecondaryActivity.class); }

ActivityComponent

@PerActivity @Subcomponent(modules = {ActivityModule.class}) public interface ActivityComponent { public void inject(BaseActivity baseActivity); // Subcomponents MainScreenComponent plusMainScreenComponent(); SecondaryScreenComponent plusSecondaryScreenComponent(); }

The ActivityComponent component provides a more generic dependency graph with commonly used dependencies (FragmentManagers, LayoutInflaters) or Context specific dependencies. As you may already know, you can get different type of the Context class in runtime and using not applicable one may cause various problems, in the event of this you must not inject the Global Context(Application) everywhere. This only one case why we may need such kind of component.

In a like manner we are declaring creator methods for sibling components.

Components creation

Owing to using Subcomponents in order to create a child component we need firstly to get a parent one. Here is a chain component creation for better understanding.

Dagger2ScopeInternalsApplication

mApplicationComponent = DaggerApplicationComponent.builder() .applicationUnscopedModule(new ApplicationUnscopedModule(this)) .build();

BaseActivity

ApplicationComponent appComponent = ((ProvidesComponent) getApplication()).getComponent(); mActivityComponent = appComponent.plusActivityComponent(new ActivityModule(this));

MainActivity

mMainScreenComponent = mActivityComponent.plusMainScreenComponent(); mMainScreenComponent.inject(this);

MainFirstFragment

this.getComponent(MainScreenComponent.class) .plusMainFirstViewComponent() .inject(this);

Nested Component Classes

Now compile the project and open DaggerApplicationComponent again. You can checkout this commit to get code that was written so far. Please attentively look through this file.

public final class DaggerApplicationComponent implements ApplicationComponent { private Provider provideCarDataRepositoryProvider; // … @Override public ActivityComponent plusActivityComponent(ActivityModule activityModule) { return new ActivityComponentImpl(activityModule); } private final class ActivityComponentImpl implements ActivityComponent { private final ActivityModule activityModule; private Provider provideLayoutInflaterProvider; // … private ActivityComponentImpl(ActivityModule activityModule) { this.activityModule = Preconditions.checkNotNull(activityModule); initialize(); } private void initialize() { // … this.baseActivityMembersInjector = BaseActivity_MembersInjector.create( DaggerApplicationComponent.this.provideCarDataRepositoryProvider, provideLayoutInflaterProvider, provideSupportFragmentManagerProvider); } // … @Override public MainScreenComponent plusMainScreenComponent() { return new MainScreenComponentImpl(); } @Override public SecondaryScreenComponent plusSecondaryScreenComponent() { return new SecondaryScreenComponentImpl(); } // … }

That is how @Subcomponents work. Forasmuch as subcomponents need to have access to the dependency graph of the parent component there is already a features available out of the box in Java world – Nested Classes. As a result a subcomponent can just reference a Provider implementation from the parent component. Please note that whenever we call a plus method a new instance of subcomponent is returned.

What about scopes?

You may wonder, why I am telling you about components and subcomponents and nothing about scopes. Because there is no such term in generated code like Scope. Try to find in generated code any occurrences of “scope”, “PerActivity”, “PerApplication”. No matches? Please try to find differences between Providers that were created in the previous example of Unscoped dependencies and this one.

private void initialize() { this.provideLayoutInflaterProvider = DoubleCheck.provider(ActivityModule_ProvideLayoutInflaterFactory.create(activityModule)); this.provideSupportFragmentManagerProvider = DoubleCheck.provider( ActivityModule_ProvideSupportFragmentManagerFactory.create(activityModule)); // … }

There is one small, but important, difference, that could be missed easily from the first sight – DoubleCheck.provider.

DoubleCheck == Scope ?

Finally, we have found how do all these scope annotations are represented in generated code. Let’s open the DoubleCheck class to what is going on there.

public final class DoubleCheck implements Provider, Lazy { private static final Object UNINITIALIZED = new Object(); private volatile Provider provider; private volatile Object instance = UNINITIALIZED; private DoubleCheck(Provider provider) { assert provider != null; this.provider = provider; } @SuppressWarnings(“unchecked”) // cast only happens when result comes from the provider @Override public T get() { Object result = instance; if (result == UNINITIALIZED) { synchronized (this) { result = instance; if (result == UNINITIALIZED) { result = provider.get(); Object currentInstance = instance; if (currentInstance != UNINITIALIZED && currentInstance != result) { throw new IllegalStateException(“Scoped provider was invoked recursively returning ” + “different results: ” + currentInstance + ” & ” + result + “. This is likely ” + “due to a circular dependency.”); } instance = result; provider = null; } } } return (T) result; } public static Provider provider(Provider delegate) { checkNotNull(delegate); if (delegate instanceof DoubleCheck) { return delegate; } return new DoubleCheck(delegate); } public static Lazy lazy(Provider provider) { if (provider instanceof Lazy) { @SuppressWarnings(“unchecked”) final Lazy lazy = (Lazy) provider; return lazy; } return new DoubleCheck(checkNotNull(provider)); } }

If you have any previous experience working with Java, you might be familiar with the technique used in the get() method. It is called – Thread-Safe Singleton. In fact, that is how Scopes are maintained by Dagger 2.

The DoubleCheck class itself is the implementation of the ProxyPattern. It wraps a simple Provider implementation with caching and threading logic, so after first call of the get method you will be provided with the cached instance. There is also Lazy Initialization logic, however this is out of scope of the tutorial.

To summarize:

• Scopes are maintained with the help of Components and wrapped Provider implementations.
• Defining custom scopes doesn’t result in changes in generated code comparing to the @Singleton annotation. All scopes are almost the same from the code perspective.
• A scoped Component only can only have dependencies of the same scope or unscoped at all.
• DoubleCheck class is not the Scope, it is just a tool for caching and providing dependencies in a thread-safe manner.
• The only one scope keeper is a developer that uses Dagger 2.

Playing with Scopes

To consolidate information, let’s break Dagger 2 scopes by improper use of library.

If you are attentive you may have noted that I used ApplicationUnscopedModule, in consequence we have the lack of the DoubleCheck wrapper around this provider. Now let’s see what are possible cases of violating Dagger 2 rules.

Recreating components

Consider the following case, when you called called dependency injection several times and haven’t checked whether component was already created.

@Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); // Inject views injectDependencies(); ButterKnife.bind(this); initUI(); navigateToInitialScreen(); injectDependencies(); } private void injectDependencies() { mMainScreenComponent = mActivityComponent.plusMainScreenComponent(); mMainScreenComponent.inject(this); }

In that case all dependencies that are in the Scope of Main Screen Component will be recreated. For instance if we add a dependency into the MainScreenModule as follows.

@Provides @PerScreen public MainScreenDependencyContract provideMainScreenDependency() { return new MainScreenDependency(); }

And call the injectDependencies method twice, two instances of MainScreenDependency will be created and only the last one will be left.

Less obvious situation may happen when you place component creation into a not proper lifecycle method, without checking existence of the component.

@Override protected void onResume() { super.onResume(); injectDependencies(); }

Activity can be hidden or showed any number of times during it’s lifecycle, for instance when the Activity is overlapped by another Activity, it could be Camera, File Chooser, etc.

Memory leaking

Another possible problem that you may encounter is not related to DI or Dagger 2 itself, but it is important to keep this in mind, especially when working with Android SDK with a large number of callbacks and listeners. Consider the following interface.

public interface GlobalEventNotifierContract { void addListener(EventListener eventListener); public static interface EventListener { void onEvent(T event); } }

For instance, we may be interested in some global application events, like push messages, etc. Implement this interface in Dagger2ScopeInternalsApplication

public class Dagger2ScopeInternalsApplication extends Application implements ProvidesComponent, GlobalEventNotifierContract { private List> mEventListeners = new ArrayList<>(); @Override public void addListener(EventListener eventListener) { this.mEventListeners.add(eventListener); } }

And in activities we are adding ourself as a listener for events.

public class MainActivity extends BaseSingleFragmentActivity implements ProvidesComponent, MainFirstFragment.SecondFragmentRouter, MainSecondFragment.FirstFragmentRouter, GlobalEventNotifierContract.EventListener { @Inject GlobalEventNotifierContract mGlobalEventNotifier; private void setEventListeners() { this.mGlobalEventNotifier.addListener(this); } }

Additionally, add the same code to the SecondaryActivity class. Run the application and try to navigate back and forth between activities. What do you think happens right now? Open Profiler and explore memory.

Memory is allocated but not released. Let’s examine what listeners do we have registered.

Something is probably wrong here :). All in all, be very careful passing callbacks and listeners, don’t forget to unsubscribe.

@Singleton != Singleton

Do not think that the @Singleton annotation magically makes your dependency a real Singleton object. This annotation due to it’s name may cause confusion. Dagger just uses this annotation for several checks. This annotation is defined in the javax.inject package (JSR 330), not in the Dagger library, that you may already met this annotation in Java.

Try to change the scope of SecondarySecondViewComponent and all related modules to @Singleton like that.

@Provides @Singleton public SecondarySecondFragment.FirstFragmentRouter provideFirstFragmentRouter(Activity containerActivity) { return (SecondaryActivity) containerActivity; } @Provides @Singleton CarPartsDataRepositoryContract provideCarPartsDataRepository() { return new CarPartsDataRepository(); }

Compile the application and open the DaggerApplicationComponent, try to find differences. Eventually, you won’t find any real Singleton stuff except DoubleCheck, Dagger 2 treats @Singleton annotation in the same way as any other scope annotation.

In conclusion, let’s look at a comment from Dagger 2 source code.

*

In order to get the proper behavior associated with a scope annotation, it is the caller’s * responsibility to instantiate new component instances when appropriate. A {@link Singleton} * component, for instance, should only be instantiated once per application, while a * {@code RequestScoped} component should be instantiated once per request. Because components are * self-contained implementations, exiting a scope is as simple as dropping all references to the * component instance.

Generally speaking, I do not suggest you to use the @Singleton annotation if you have just started working with Dagger 2, it will be much better if you try to create your own scope and learn more about scopes instead of blindly using @Singleton everywhere.

Conclusion

I hope that after reading this article you have got better understanding of Scopes and what stays behind the scenes of Dagger 2 dependency injection.

Dagger 2 is a great tool that will make you application much more flexible and maintainable if applied correctly. Don’t hesitate to look into internals of implementation that will help to use a tool in a right way.

If you have any questions or suggestions please leave them below in comments.

Source code

The post Understanding Dagger 2 Scopes Under The Hood appeared first on Oleksandr Molochko.

]]>

Xdebug is a great way to eliminate var_dump(), print_r() statements and make your PHP debugging experience better and more productive. When you have a simple application there is not really need to spend time on configuring Xdebug, you can go along with dumping varialbes, however in such applications and frameworks like Magento it is really hard to debug using var_dump statements. This is caused by a huge code base of the Magento platform as a result use of caching everywhere where it is possible or not. Furthermore, tracing a path of a function call can be a really tedious task. Using Xdebug together with PHPStorm makes debugging much easier.

Introduction

There are lot of articles out there, however I remember the time when I have tried several times to configure Xdebug, unfortunately all attempts failed because of the lack of understanding how all these tools work under the hood.

The main intention of this article is to explain how does XDebug work and what problems I’ve encountered while configuring it for my projects. Personally, I always try to understand how something works under the hood and I think that worth it in any case.

I won’t cover all benefits and features of XDebug, there is enough information already provided. One important reason to start using XDebug is debugging in production. In case of using var_dump(), print_r() statements you are not only providing visitors a ton of scary information, but you are also risking to have your sensitive data be exposed. For instance database credentials.

Understanding Xdebug

First of all we need to understand how does the XDebug debugger work. Xdebug is a PHP extension, that is loaded dynamically as a shared library (.so), therefore before using it you need to compile it. We will see later how to compile and install this extension.

XDebug differs from other debugging tools you may be familiar with. XDebug works over the protocol that requires your local machine to listen for incoming connections from a server where an application you are debugging is located. You may already have used debugging tools that simply connect to a remote server or a process of your application. But in case of XDebug it works other way round, it is similar to the concept of “callback” in programming languages. Lets explore this communication step by step.

XDebug connection flow

There is a nice illustration that will help to understand the flow better.

• Start a debugging server on the local machine. Usually port 9000 is used to listen for incoming connections
• Send the HTTP request (with some arguments and parameters) to activate XDebug on the server side
• Depending on the configuration, XDebug on the server side will try to connect to your machine to start debugging session. Commonly the host is set to localhost:9000 or the IP address from which the request was made
• A debugging session started and works further over the DBGP protocol

To sum up and get better understanding here is one more useful illustration for the flow.

If you are curious, as I am, about these steps, how everything works under the hood, let’s explore them deeply but without concrete configuration for now, I will show a real configuration a bit latter.

XDebug client/server/listener

As I have already mentioned, XDebug works in the way that the part that wants to debug application should listen to incoming connections.

Let’s follow the naming convention and call the guy that listens for back connections from a remote server – XDebug Client. Generally this may be called a server, as far as it is listening for connections from a remote server.

Consequently, on your local machine you need to start the XDebug client on your local machine in your code editor. I prefer using the PHPStorm IDE, but you are free to use any other editor, like Sublime Text 3, Brackets, etc. But at first ensure that your editor already has an XDebug plugin available.

Later, I will explain how to configure PHPStorm, but the final result should be the same, you have to have the XDebug client listening to a port, usually it is 9000. As a result you should be able to verify it for instance using the following command on Linux.

Or in case you are a Mac user the following command can be useful.

In my case I got the following output.

You may have a question regarding this matter. How to listen for incoming requests if you have a dynamic IP address or your network behind a firewall. The answer is to use SSH port forwarding, we will see later how to forward ports.

Activating the XDebug session

To start debugging, considering that a server needs to connect to our machine, we need to inform the server about our intent. This can be done in the several ways:

• Append the GET parameter or the POST variable – You can append the following parameter XDEBUG_SESSION_START=name to your website url, for instance http://example.com?XDEBUG_SESSION_START=PHPSTORM. In that case the server will issue a cookie that will be set in response.
• Set the cookie manually – You can set the XDEBUG_SESSION cookie manually to enable the debugging session. Of course setting cookies manually is a tedious task, so there are some plugins available for that purpose.

Here is how it looks like in the request headers. You can see that the XDEBUG_SESSION cookie is being sent with the request.

To disable debugging session you have to delete the cookie.

Setting up XDebug

As the title states, in that article I will show how to debug a Magento project, however you are free to use any kind of PHP project. Magento is the CMS that really requires debugging, because of enormous codebase and not trivial request handling.

Configuring XDebug server

I won’t show how to install the PHP interpreter and assume that you already have PHP up and running.

In my case I have the following configuration:

As it was already mentioned, XDebug is a native extension for PHP and provided as a shared (.so) library. Hence, you need to compile and install this extension at first. The most easy and fast way I have found is to use PECL.

To install XDebug using PECL type the following command in the terminal.

After that you will see compilation process and when it is done you should see the path of the compiled extension. On my server I got the following output.

Next, you need to enable the XDebug extension by modifying the php.ini configuration file. In my case I had to create a config file located at /etc/php/7.0/fpm/conf.d/20-xdebug.ini. But configuration options, of course, are the same.

The XDebug extension has a lot of configurable options, they are described in details on the official page. For my needs I used the following configuration.

I have to explain some of the options above.

xdebug.remote_host – is set to localhost, as far as we will use port forwarding to our local machine further. You can set it to a specific domain or a IP address.

xdebug.remote_connect_back – is disabled by default. This option allows to dynamically connect to an IP address that requested start of the XDebug session. But this option won’t work in most cases, as far as mostly your machine will be behind firewalls, have a dynamic IP address and ports closed. Considering all these problems, the best option is to use port forwarding.

xdebug.remote_port – I have used not a default port number, in order to make it more explicit when we will configure port forwarding.

xdebug.idekey – The value of this option defines the key that should be pass while initializing the XDebug session as was previously discussed. In my case the value is PHPSTORM.

Other options are self-descriptive, and for additional information you can refer to the official page.

In on order to apply changes restart the PHP service, if you have the FPM type of course. Now verify that the XDebug extension has been successfully enable, for instance using a well-known function – phpinfo(). You should get a similar output.

Configuring XDebug client

Now it’s the time to configure our IDEA, PHPStorm in the context of this article. Before starting, please make sure that you have an exact copy of your files on your local development machine. This is required for a correct work of the debugger.

Open your project. Open settings in the IDEA and search for xdebug. Set the following options according your requirements. Please note, that even that we have set the port number to 9900 on our server, however we have 9000 on the local machine. You will see in a while why. But please check twice that there are no local process already using 9000 port. If you have PHP-FPM locally installed it will use this port, so you have to use another one.

Now search for DBGp and set the settings according to the previous configuration. This configuration is required for multiuser debugging. If you are the only one person debugging the application, you may skip this step.

One more thing you can configure is to create a Remote Debug configuration. I have set it up as follows.

You can run this debug configuration to start listening for configured port, however personally I haven’t found any difference between this configuration and the regular configuration.

Start Debugging and port forwarding

Finally, we are up to start debugging, but before that we need to configure SSH Port Forwarding. I won’t explain this concept deeply, you can find a lot of articles about port forwarding.

However, for better understanding here is a picture how it works.

So basically you are redirecting all requests from you server that come to port 9900 to the local machine on port 9000 through the SSH Tunnel. To setup the forwarding you can use the following command.

Ensure that you have no error messages, for instance if on your remote server the port is already used, the port forwarding fails. In my case I had the following message Warning: remote port forwarding failed for listen port 9900

As you can see, the remote port is set 9900, but our local port that PHPStorm debugger listens to is 9000. You can freely change any of these ports by changing the configuration explained above. Such type of communication is firewall sustainable and much more secure.

To init the XDebug session, as was mentioned earlier, you need to set the cookie. To do that there is a great extension for the Chrome browser – XDebug Helper. It do this work for you, also it is capable of tracing and profiling.

Now set breakpoints on the line you are interested in. For example I will a breakpoint in the pub/index.php file.

To start listening to incoming debug connections. You can do this in several ways. You can just click on this button in PHPStorm

Or you can use the preconfigured remote debug configuration.

After all these steps enable the extension in your browser on the web page of your application, by clicking on the green bug icon.

Finally, refresh your page and after some time you should see a familiar debug window appeared in PHPStorm. Also you may be asked about accepting debug connections.

Now you are able to debug your application step by step without a single var_dump

Conclusion

In that article I tried to explain how does XDebug works, I think this is crucial for using it. Furthermore, it is always worth it to understand how things work under the hood.

The configuration described in that article is only one possible way, that works well in my case. If you have other situation, you may need another configuration. Anyway, if you have problems with configuring XDebug, please feel free to ask below in the comments.

The post Understanding and using Xdebug with PHPStorm and Magento remotely appeared first on Oleksandr Molochko.

Overriding modules is a really powerful feature of the Prestsahop CMS. It makes possible to change behavior and appearance of modules without altering original source code. However, due to latest changes in Prestashop 1.7 a lot of changes were made and some features are gone forever. Therefore this article is intended to explain the most widely used cases and understand how does the Override system work in Prestashop and what overriding techniques are still available.

Prestashop 1.7 changes

One important note is that starting from Prestashop 1.7 overriding is considered a bad practice. According to this article, overrides still be available, even so a lot overriding features are already missed in 1.7.X versions. For instance you cannot override core classes anymore by putting your classes in prestashop-root/override/classes/module/Module.php like that.

Only classes for translation can be overridden this way. Have a look at the getFileToParseByTypeTranslation method in AdminTranslationsController file

Prerequisites

Before starting any modification and creation of source files enable Debug Mode from your Admin Panel. Advanced Parameters -> Performance -> Set Debub Mode to true . Also you may need to enable recompilation of template files in case you are going to change layout. Alternatively, you can delete the cache/class_index.php file

Throughout this article I will be working with a default module Main menu. Of course you can apply all these techniques for other modules.

Documentation vs GREPping source code

When you are trying to find how to implement something using a CMS, usually you refer to the official documentation. But, based on my personal experience it may take much more time trying to find an explanation of some part of an engine by googling or reading the official documentation. For that reason I prefer grepping source code. The most valuable benefit you get by using this method is understanding how everything works under the hood, what architecture is used, how some feature is implemented. After exploring several software products, you will get an idea how something could be implemented, what are pros and cons, as a result you can use acquired knowledge while building your own software products.

Override system

In order not to break most of available modules, Prestashop 1.7 still supports overriding for modules. As far as we are interested in overriding modules behavior, I encourage you to have a look at the Module class that is located at prestashop-root/classes/module/Module.php.

Override Template Files

At first let’s find out how to override a module’s view appearance. Find the _isTemplateOverloadedStatic method

As you can see from the code above there are different possible patterns that can be used to override a module’s template files. In my case I have created a file at the following path prestashop-root/themes/my-theme/modules/ps_mainmenu/ps_mainmenu.tpl.

Override Module Classes

To modify behavior of a module’s class you have to put your modified class at prestashop-root/override/modules/NAME_OF_MODULE/class_to_override.php. And your class should have name defined according to this pattern {NameOfOriginalModuleClass}Override. For example override/modules/ps_mainmenu/ps_mainmenu.php and the class name should be Ps_MainMenuOverride. To understand how does the CMS process module classes overrides, find the coreLoadModule method.

As you could see the implementation is quite straightforward. And according to these rules our class might look as follows.

Conclusion

All in all Prestashop 1.7 has breaking changes for a developers who leveraged the overriding feature extensively. According to the prestashop blog posts it seems that the team want to improve an architecture of the system, this is definetly commendable, but any major changes should be added step by step leaving backward compatible options for developers.

If you are still trying to find a way to override the CMS core classes without direct modification of source code, you could probably find a workaround, for instance:

• Use advanced PHP features like runkit/classkit.
• Explore the request flow of the Prestashop. And add your own logic for loading classes (for example modify autoload.php). But every time you update the CMS core, you will need to add/review your custom implementation. Here you can find a detailed image of a request processing in Prestashop.
• Explore source code of the CMS in order to find other ways like runtime inejction/modification via include statements, etc.

I hope this article will be helpful for understanding how override features works under the hood, if you have any questions please feel free to leave comments below.

The post How to Override Module Templates and Classes in Prestashop 1.7 appeared first on Oleksandr Molochko.

Is there any reason to use Twig with WordPress? Do you like PHP code snippets inside your HTML layout templates, can you open any large template and find a part you are interested in within a few seconds? If you can, you probably have a small template file.

When I first started using CMSes written in PHP, I was unpleasantly surprised by the way they produce HTML code. I tried one CMS and then another, however found the same situation, HTML and PHP code snippets were tightly coupled and mixed in a chaotic way. If you have a template file containing less then 200 lines of code it may seem not so scary, but in case of large projects you may end up spending hours trying to found unclosed tag, bracket or quote. Furthermore, WordPress has introduced a really bad practice – opening a tag in one file and close it in another one.

Are PHP and HTML best friends?

I have noticed that sometimes newbie developers don’t understand how does the HTML/CSS/JS frontend trio relate to “server side languages” like PHP. Why do I put the server side word into quotes? Because, to be honest there is only one thing that makes a language to be a server side, is ability to handle requests coming from the client-side. On the low level it means that a language has built-in features for opening sockets and handling requests of a specific protocol, in our case HTTP. As you may already be aware about Node JS, this a good example of a language that initially was used only inside a browsers’ virtual machine. We have a standard specification called ECMAScript. Now it i implemented on the server side and used quite heavily for different kind of applications. NodeJS has a great asynchronous I/O and request processing.

Why I am telling you this? In order help you grasp an important idea. There are a lot of different languages that can be used for producing HTML code, they all have different syntax, features and anytime you mix frontend markup languages your are cornering yourself into boundaries, since you will have to change a lot of code if the backend technology changes. To avoid such problems you have to keep your templates (layout) files decoupled from your server side technologies. Template engines usually use some sort of DSL that is processed by a template engine itself. In future articles I will show how can you create a web site (powered for example by WordPress) from HTML layout without making a single change in HTML files.

Why use a template engine?

• Decoupling – As I have already mentioned, decoupling is a great way to make your applications more scalable and extendable, since you are keeping different concerns separetely.
• Speed and Efficiency – Development becomes much faster and efficient when a template engine is used, I know you are already tired of openning and closing PHP tags inside your templates for just a single echo statement. Of course compilation of templates requires extra processing time, but it’s worth it.
• Convention – A template engine force you to use some convention and you usually don’t have a choice to implement something quite differently.
• Readability and Simplicity – Typically, a template engine’s syntax is much more readable than code snippets. That saves a lot of time while development and debuggings.
• Security – While writing echo snippets or similar constructions you have more freedom, I have seen a number of time, when devlopers put everything inside a template file starting from core business rules (like calculating salary) to raw SQL queries. When you are using a template engine you at least will have to find how to implement your dirty idea, but I hope you won’t find and end up with moving all non related thigns to a proper layer. In addition some template engines provide security features like sandboxing.

Meet Twig

Twig is a template engine for the PHP programming language that is built to boost your development experience and save a lot of time and spare your nerves. It will bring new advantages in your development process and help to eliminate spaghetti code. Twig is becoming even more popular and widely used, for instance, Twig is the core template engine in Drupal 8. Opencart starting from 3.0.0 version is also leverages the Twig template engine. I guess that WordPress starting from 5.0.0 also will use Twig or some other template engine.

Lets have a look at some sort of before-after example.

Without using Twig.

The post Efficient, Fast and Scalable WordPress development with Timber (Twig) appeared first on Oleksandr Molochko.

TP-Link WiFi Router WR841ND is a very popular router because of it’s price, especially in my country. However this router is able to provide much more features than the stock firmware has. One day I decided to make it more powerful and feature-rich, furtermore I noticed message on the official tp-link page about possibility to install a custom firmware on your own risk. Therefore I decided to install the DD-WRT firmware. Here is one possible usage of DD-WRT firmware.

I flashed the firmware successfully and everything seemed to work, but than I recognized that I had to restore all settings (like port forwarding, mac addresses binding and so on) manually. Fortunately, I had backed up the config file. But this binary file is suitable only for the stock firmware. I had no time for exploring and parsing the binary file, so I decided to rollback in order save my settings in a plain text file and than flash the DD-WRT firmware again. I successfully rolled back, saved my settings, but while updating “web flash firmware” in the DD-WRT web gui, something went wrong and my router got bricked. The router started blinking all LEDS periodically, that is also known as the Boot Loop. The router’s internal interface went up and down periodically as well.

Is this the end?

Don’t worry if you bricked your router, there are a lot of ways to get it back to life. Some of the mostly used methods are:

1. Communication over UART (TTL). In this case you need to disassemble your router, solder some wires to the pins on the routers motherboard and than you will be able to connect to the router over the UART interface. This is usually done with USB<->TTL converter.
2. Using built-in TFTP auto-unbricker . Some routers have a built-in TFTP client running and looking for a TFTP server in the local network to download a firmware and debrick itself. This feature is usually provided by U-boot.

I could easily solder some wires and connect router over UART, but I really didn’t want to tear down my router, so I decided to use the second option.

Unbricking the router

All tutorials that you could find on the Internet suggest to use constant IP addresses like 192.168.0.68, generally it could be any address. It depends on a boot firmware flashed to your router. Therefore, you should try each IP address from the range from 192.168.0.2 to 192.168.0.68 until you find a proper one…. Furthermore, the router’s TFTP client will look for a file with some hardcoded name, that could be different for every firmware. Of course you shouldn’t try every possible combination, this article’s intention is to help you find exact parameters.

We will find out exact IP address and file name using network sniffer – Wireshark. Using Wireshark we can see everything that happens in our network, all packets going back and forth. Here are the steps to unbrick the router, that helped me.

1. Download Wireshark from the official site and install it.
2. Than run Wireshark select an Ethernet Adapter which you are going to use for connecting the router via a UTP cable.
   
3. You should see an empty log list at first.
4. Now its time to connect your router. Connect the UTP cable one side to any LAN port (not WAN) on the router and other side should go to your PC or laptop.
5. Power on the router after the cable is connected. You should see packets appearing in the log list.
   
6. Clear the log list in Wireshark. Power off your router. Than hold the WPS/Reset button and power it on. Keep holding the WPS/Reset button for some time(about 10 seconds).
7. If you are lucky have done everything right you will see an ARP packet like that. 19 1.389742 ba:be:fa:ce:08:41 Broadcast ARP 60 Who has 192.168.0.66? Tell 192.168.0.86
   
   As you can see from the line above in my case the router is looking for 192.168.0.66.
8. Let’s configure our network interface to have the IP address that the router is looking for.
   
9. Enter the IP address you got from the Wireshark and subnet mask 255.255.255.0.
   
   Save settings.
10. Finally, we need to find out what file name is the router looking for..
11. Start Wireshark again. Power On the router holding the WPS/Reset button. Wait for a few seconds, you will see the log list growing. Look for a TFTP packet (you can find it by sorting the protocol column). You should find something like that 159 7.304062 192.168.0.86 192.168.0.66 TFTP 85 Read Request, File: wr841nv8_tp_recovery.bin, Transfer type: octet, timeout=2
    
12. This packet contains the file name the router is looking for. In this case it is wr841nv8_tp_recovery.bin. There’s a little left to do now.
13. Next download the TFTP server, for example that one and install it.
14. Now you have to find a recovery file for your router. Go to download section and find Firmware WITOUT Boot section, usually this is stated in the file name or description. In my case I will use the OpenWRT firmware downloaded from https://downloads.openwrt.org/chaos_calmer/15.05/ar71xx/generic/ the file name is openwrt-15.05-ar71xx-generic-tl-wr841n-v8-squashfs-factory.bin
15. Rename the downloaded file to the value you got from TFTP packets, in my case it is wr841nv8_tp_recovery.bin in my case. Run TFTPd as administrator and select a proper IP address and directory where your firmware is located.
    
16. Now power off the router. Connect it to you computer via the network cable. Hold the WPS/Reset button and power it on. Wait until a progress dialog appears. After that you will see progress in the TFTP Server Tab and wait until it completes. Do not power off the router give it some time to update firmware and boot up.
    
    I have tried to do this on several PCs. Firstly on a Windows 10 laptop, an XP Laptop and the only machine that succeed to work was a Windows 7 desktop PC. It doesn’t depend on type of a computer, but mostly on software installed (especially firewalls and anti-viruses software). So if you have TFTP request from the router, but it doesn’t upload firmware (you don’t see the progress dialog) then try to use another PC. You can use any OS, a TFTP Server implementation is available for any OS (macOS, Linux, Windows).
17. Finally reset your network adapter settings to Automatic or your previous settings. After that you should be able to access the router’s control Web UI as usual using browser (192.168.0.1).
    

Conclusion

In this article I’ve shared my experience of debricking my router. The main idea of the article is to show how the recovery process works under the hood and how can you find out the required IP address and file name, that your sick router is looking for. Hope this article help you to fix your router. If you have any questions, please feel free to leave them below in the comments section.

The post How to Unbrick TP-Link WiFi Router WR841ND using TFTP and Wireshark appeared first on Oleksandr Molochko.

The Clean Architecture is the term proposed by Uncle Bob, that refers to principles and design practices used for building an architecture for software. It is defined in more abstract way, causing a lot of questions and debates.

This article is intended to explain the most important concepts of The Clean Architecture. Unfortunately Fortunately, this will not be a step-by-step guide. I think that such kind of guides are only applicable in case of technical questions, that don’t require much thinking about a step to be done. All architectural decisions should be well-considered. Some of concepts described in this article may seem absurd at the first glance, however they should make much more sense after you adapt them in your project.

Introduction

The main idea behind the Clean Architecture is quite similar to architectures and concepts described in the previous chapter (Hexagonal, Onion). I would even say all they about the same. Generally, it is just a set of the most strong and important ideas from preceding architectures. Don’t be naive to assume that the Clean Architecture is the silver bullet. Even if you have grasped the ideas, it doesn’t mean that you could apply it everywhere and result in a dramatic codebase improvement and a project success. Using it without solid understanding why do you need it, just because this topic is ubiquitous and everyone tries to apply this architecture, may lead to even worse results than it would be without applying the principles.

How should a good architecture look like?

Before we begin exploring the Clean Architecture, we need to understand why do we need it at all and what features should a good architecture have ? What requirements we expect to be fulfilled by an architecture, and generally what do stakeholders and business expect from a software system?

1. Ability to respond as quickly as possible to business changes, making a system competitive.
2. A system should be divided into modules separated by boundaries. Any module, detail, delivery mechanism should be swappable, without downtime at best.
3. A system should be maintainable, easy to understand, extend and deploy. This will dramatically increase programmers productivity.
4. A system should be scalable. An ability to grow horizontally to handle higher loads and big data amounts.
5. A software system should have a strong foundation that allows to defer some decisions about implementation details, delivery mechanisms, libraries, etc.
6. An architecture should represent and support the intent and use cases of a system.

Inversion of Control. The Dependency Rule

One of the most important and ubiquitous concept that is used almost in every framework is Inversion Of Control. You almost already have heard about this principle, for example working in Java EE world with IoC containers.

One of the most essential ideas in understanding The Clean Architecture is The Dependency Rule. It states that

The rule doesn’t directly and tightly related to the concept of Inversion Of Control, however applying the Dependency Rule forces us to apply IoC. And that is good point, let’s find out how is it applied in the Clean Architecture by examining the flow of control. You can find the following diagram in the right bottom corner of the initial architecture diagram.

What is it all about? We will transform the diagram a little bit for making more sense and ease of understanding.

Let’s follow this flow step by step. Consider some action like a button click occurred. Don’t pay much attention to different concepts used further, they will be explained in greater details later in the article.

1. A button click event is handled in a controller. The controller triggers some method of an object that implements the UseCase Input Port interface.
2. We have just crossed the boundary between two layers and are in the UseCase layer now.
3. A use case implementation processes the request by orchestrating entities or other domain core objects.
4. After getting results from entities the use case implementation invokes a method of an object that implements the UseCase Output interface.
5. A Presenter gets result from the Use Case and transforms it to a proper shape followed by passing it to the View layer.

You may also wonder what do these arrows represent? In short, they could be described as follows.

1. Implements an interface.
2. Uses an interface implementation, composition, has-a relation.

One more type of arrows is a dashed one. These arrows shows a real flow of execution, from the programming point of view it could be represented as a stack of function calls or movement of the Program Counter (PC).

To make a final point in understanding this flow let’s look at the possible code implementing this concept. First of all, let’s define a use case, in this case it will be AddProductToCartUseCase.

In the code above we are passing AddProductToCartOutputPort to the execute method, however it could be implemented in other ways, for example the implementation could be injected as a class member. It depends on a concrete project, language, approaches used, etc.

Next we will define AddProductToCartInputPort and AddProductToCartOutputPort interfaces. Please note the package name where all these interfaces are defined.

A Presenter and a Controller might look as follows.

Herein AddProductToCartInputPort and AddProductToCartOutputPort are injected in the Controller class, however it could be implemented differently, as have been already mentioned. However, using both controllers and presenters may seem weird, this is a contentious question, the primary goal here is to understand the flow.

In real projects a Presenter and a Controller are usually combined, but a View should be a separate guy. Please also note that I am not refering to the Controller from the MVC pattern and to the Presenter from the MVP pattern respectively. They just represent a general intent: to control and to present.

Furthermore, such naming conventions probably should be avoided.

The main idea of the described above is that higher level policies should not depend on details, frameworks, UI, etc. As you may have noted, our AddProductToCartUseCase operates only with interfaces that are defined on the same level as the use case itself. Here comes the power the Dependency Rule represented by inversion of control, dependency injection, dependencies inversion. We can swap an implementation at any time, it should only conform to the defined interface, so our high level rules and policies are isolated from the things that are constantly changing, making the application core stable.

Layers and Circles

It is the time to understand this diagram. From the first sight it might look weird, because most traditional architectures are defined from top to bottom. Let’s understand this diagram, but before exploring it, it won’t hurt to show the flow of execution on this diagram.

You may wonder what does this curved line represent and why does it pass boundaries back and forth several times. This is just a possible flow when you have a use cases that requires multiple details to perform some action. The term details in this context means any tool (library, db, delivery mechanism, framework, device) that exists on the outermost layer.

For example, when an order is being confirmed, this action requires a lot of steps to be done, like generating invoice, adding history, notifying managers, checking product availability, whatever. Therefore the code execution flow will look something like that curved waved line. Please note that this line doesn’t represent dependencies relationships, dependencies must only point inward like that.

Finally, I will transform the initial diagram to a more common shape for understanding.

This flow is almost present in any application.

1. Some external action is occurred, for example a user clicked on a button.
2. This event is passed from the View layer to the Presenter layer. A presenter class is tailored for converting data to a proper representation and passing it to a use case class.
3. The main duties of the Use Case layer is to control and orchestrate entities and other domain level objects according Application specific rules. But before that domain objects should be created based on data stored in a database or somewhere else. This is usually done through gateways. In this case this the Data Store interface.
4. After entities are fetched in the Use Case layer, a Use Case implementation orchestrate entities and domain objects according to Application Specific rules.
5. On the diagram I have used the Aggregate root as an entry point to the Domain Core Layer. This term is defined by Domain Driven Design principles.
6. When the request is processed according to business rules of an application the result is passed back to the UI layer for example, if required.

I hope these diagrams will help you to understand the Clean Architecture approach better. Now let’s examine each layer separately. We will start from the outermost ring and then will move step by step to the core layer.

Details. Databases. Delivery Mechanisms. UI. Web

This is the outermost layer on the diagram. It will be probably the largest layer of any system, since there is a variety of devices, libraries and frameworks. They are constantly changing, therefore we should abstract high level (inner layer) policies from details. Let’s go through some of them.

1. Network. HTTP, API, Websockets, Sockets. Basically whole the OSI model or any other alternative are just delivery mechanisms.
2. UI. Web UI, Native Window UI, Command line, API, ViewControllers, Activities, Fragments. All these examples are just UI mechanisms and therefore details.
3. Databases. SQL, NoSQL, in-memory, file-based databases. Databases are just persistence or caching tools.
4. Libraries and Frameworks. There are lot of examples could be listed here. We should always try to abstract our core layer in application from libraries, however it is not always possible fully, since some libraries may be crosscutting.
5. I/O devices. Input devices (touchscreen, keyboard), persistence devices (NANDs, HDDs), output devices (screen, speaker). Even the Linux Kernel written in C uses the file abstraction for all I/O devices that makes the kernel really powerful and portable.

I guess thats enough examples, the main idea should be clear as far as things on this layer change very frequently we should be protected from this changes by the wall of abstraction and inversion of control. This will not only make a development process much easier, but will also isolate possible bugs. Furthermore, this layer is usually hardly testable because of its changing nature and instability.

Presenters. Controllers. Gateways. Interface Adapters.

This circle layer is named as Interface Adapters in the original article. That means that all data that is passed to outer or inner layers should be transformed here to a convenient structure for a layer being passed to. For instance, data that is passed to a View primary should contain only String fields, so the view could just display it without any extra work.

You should be already familiar with such design patterns like MVP, MVC, MVVM. I won’t explain each of these design patterns. The concrete one is selected according to project requirements, type of a project and other factors. This layer is just a glue between the outermost details level and the application layer, in that case represented by the UseCase layer. Roughly speaking, the letter M (Model) in the Clean Architecture is embodied by the Use Case layer, but not by Entities or other Core Domain objects. Or the Model could be just the data passed back and forth between the UseCase layer and the Interface Adapters layer.

The next guy that lives on this layer is the Gateway. Generally gateway is just another abstraction that will hide the actual implementation behind, similarly to the Facade Pattern. It could a Data Store (the Repository pattern), an API gateway, etc. Such as Database gateways will have methods to meet the demands of an application. However do not try to hide complex business rules behind such gateways. All queries to the database should relatively simple like CRUD operations, of course some filtering is also acceptable.

This and any further layers should be testable in contrast to the Details layer, that is usually is really hard to test.

Use Cases. Interactors

Now things start to get interesting. The Use Case layer is not as trivial for understanding as already discussed layers. And very often there are a lot of misunderstandings about the purpose of this layer.

A Use case is a list of actions and communication steps between a role and an automated system that are required to achieve a goal. This definition contains some important points that are worth detail investigation. A set of use cases is how a user sees a software system from the functionality perspective.

I guess that everyone is aware of the Use Case Diagram. It is used on the first steps of a software system design. The more defined use cases cover possible system operations, the easier it will be to design a proper architecture and choose a development process strategy and methodology.

From the technical point of view, that we are most interested in, a use case mostly is an orchestration of Entities. Entities will be discussed in a while, for now you need to know that they contain Critical Business Rules and dance to the tune of use cases, but according to the business rules. Also there is a term called Interactor. A Use Case and a Interactor are related terms and usually used interchangeably. I think, it would be better to say that an Interactor object implements a Use Case of a system.

What does application specific mean in that case ? Application specific rules can be changed if application requirements change. An application in this case is an automated system. For example consider a bank as the core business set of rules. An ATM application will have it’s own rules, a web system for personal account management will have other rules, a mobile application also may have different rules. That’s why these such rules are application specific.

I have found a great use case example in this article. This is a use case of withdrawing money from an ATM.

I thinks that is a really good example of separation between Application Specific Rules and Critical Business Rules. Critical Business Rules that belong to the Entities layer are bolded in the use case scenario above. Personally I would put one more step to the business rules, however the idea should be clear.

Usually a use case have some input data required and output result is returned when a use case completes. The result can be represented not only by data directly returned, but a deferred event like a callback function call or just some indication of success. Therefore let’s modify slightly the use case, adding input and output data.

As you can see I have added input and output fields to the use case description. They are specified more from a user’s perspective, not as arguments and the return value for the execute method from an interactor implementation class (for instance).

Usually a use case is a single atomic action. Sometimes developers are mixing multiple related use cases into a single interactor/use case class (as I have done here). This is done to eliminate boilerplate code, however this approach violates Single Responsibility and Separation of Concerns rules. Personally I prefer to keep use cases separately.

Entities. Domain.

The last and the most important layer from the architecture perspective is the Entities or the Domain Layer. Entities encapsulate Critical Business Rules. Critical Business Rules are rules that support a business existence, but not necessarily a software system presence.

The last and the most important layer from the architecture perspective is the Entities or the Domain Layer. Entities encapsulate Critical Business Rules. Critical Business Rules are rules that support a business existence, but not necessarily a software system presence.

This layer should be the most bulletproof and external changes shouldn’t affect operation of this layer. Entities usually embodies business rules that even make sense even without any software system or application.

From the technical side an Entity is an object that mostly contains methods and logic, not just data. Entities are not DTOs, DAOs.

Designing the Entities layer is not always a trivial task. In case you are developing an enterprise application you will probably spend a lot of time architecting this layer and errors this layer may have really serious consequences. There is a set of really valuable rules called Domain Driven Design. These concepts aims to ease development of large enterprise applications with complex business rules. In case you are not developing an enterprise application, these concepts still are worth your attention. Therefore I encourage you to study the Domain Driven Design approach.

What about simple applications that are not a part of enterprise ? The Entities Layer should contain most general and high-level rules and policies that should not be affected by the changes in upper layers.

Entities must not persist their own state in a database. There are a lot of ORM libraries and framework that impose direct mapping of Entities to the database, however such approach violates the Dependency Rule. For example when a base Entity class extends some Database model class it is still aware about database and other low level details. Some frameworks apply Inversion of Control to solve this problem, other use bytecode weaving, generally speaking Entities should not be aware about existence of a database.

Entities vs Use Cases

One tantalizing question is the difference between Use Cases and Entities layers. When to use one over another ? Here is some comparison between two layers as I understand them.

1. Use cases contain application specific rules, on the other hand Entities contain core business rules.
2. Use cases usually don’t make much sense without an application, but Entities are the rules and policies that may exists without any automated system.
3. Entities contain rules that rarely change and not affected by changes in the outer layers.
4. Use cases orchestrate the flow and operation of Entities. All data that is required by Entities or the Domain Layer usually passed through/by the Use Cases or Gateways.
5. When application requirements change, at best there should not be any modifications made in the Domain Layer (Entities), but most of changes will happen in the Use Case and Details layers. However, when some core business rules are modified then these changes will be mapped on the Entities layer.

Conclusion

In this article I’ve tried to share my experience of understanding and applying the Clean Architecture principles. All described rules and concepts may be really valuable if used correctly. Otherwise they may cause much more mud and spaghetti in your project. Therefore novice developers should get enough experience before applying these rules.

These rules should not be strictly followed, I would say that the Clean Architecture is just a set of practices that should give an idea about further decisions and design steps regarding a software shape. Every project requires a unique approach that should be made by an architect based on a lot of factors.

If you have any questions regarding the article, please feel free to contact me or post comments below, discussion is always a great way to reach a consensus.

The post Clean Architecture : Part 2 – The Clean Architecture appeared first on Oleksandr Molochko.

This will be a really short article explaining how to download image from the browser side while running tests using the WebdriverIO library.

The problem of downloading images

I have a use case scenario, that requires to verify that a loaded page has the expected image. The main problem that in order to save image we have not so many options:

• Save using the system save dialog. The problem is that we are not able to control system dialogs using the Webdriver API
• Get the url and download it locally. This will not work if an image is protected and requires cookies or any other kind of authorization
• Execute a script on the browser side that will download an image, converts it into a transportable format and finally sends the result back. However this method also not without its difficulties.

Async nature of Javascript

The only one method I have found working for this case is the third one. But the issue here is that network communication, IO operations are executed in an async manner in order not to block the UI thread. Let’s start by creating a simple script that will just download an image by a URL and convert the downloaded image into a Base64 string.

You can check this function, it should print a base64 string representing an image to the console and send the result back to the callback if it was provided.

As you can see there are two async calls in the function, the first is the XHR request and the second one is reading blob data using FileReader. That’s why we cannot get the response immediately.

The JSONWire protocol

I think it is always worth knowing how do things work under the hood. All communication with between client libraries and browser drivers is done via the JSON Wire Protocol. Therefore all data is sent in the plain JSON format. This can help you to understand how all this works internally and debug issues. You can use the RequestDebug library to log all request coming back and forth.

The executeAsync call

The webdriverio library provides two methods for executing scripts on the browser side, execute and executeAsync. The former function is synchronous, so the result of evaluating the script is returned to the client after the script execution is finished. In our case it this function won’t work because of async calls in the downloadImageToBase64 function.

The executeAsync function is exactly what we are looking fore. The last argument of the function is a callback function that should be called from a script being executed on the browser side when it completes. Before we can use executeAsync we need to set the script execution timeout. According the documentation it should be set to 30 seconds by default, however in my case it doesn’t even wait for a couple seconds and the following error is thrown: Error: asynchronous script timeout: result was not received in 0 seconds.. You can set the timeout as follows.

We need to inject our function in the context of a browser used for running tests in order to use it in a script later. Of course you can just pass all functions in a single executeAsync call, but I would prefer a bit cleaner way. In my case I need several helper functions that are used on the browser side, so I have defined a custom command that sets all these utils/helpers functions into the global (window) context of a browser.

Finally let’s create a custom command for wrapping the downloadImageToBase64 function.

Downloading images using Webdriver

Now it is time to test our code. Personally I am using Selenium docker images, or rather the Selenium Hub image with multiple worker nodes. Docker always helps me to keep my machine clean. As a result in my case I am using remote connection. Here is the working example for downloading an image in the Base64 format from amazon website.

Please note that this image is hosted on the other domain than amazon, therefore by default we are not able to execute the XHR request because of the Same-origin policy. Appropriately, you may need to disable security features of the browser. In the event of the chrome driver I have the following configuration.

Now you can try to download a book cover image by a url like that.

In my case I got the following base64 image. Open Developer Tools and get the src attribute of the image.

Here is an utility class that you may need to save a base64 encoded image to a file.

Conclusion

In this short how-to guide I shared my experience about dealing with image downloading using the WebdriverIO library. Nonetheless, the main intention of this article is to show how you can use custom commands and scripts execution on the browser side to get data from a page. If you have any troubles or suggestions please leave comments below.

The post How to download images using WebdriverIO (Selenium Webdriver) appeared first on Oleksandr Molochko.

DD-WRT is a great firmware that is developed to enhance the performance and bring powerful features to cheap routers (even < 50$), making them super routers. However one feature is missing by default is transparent proxifying of network traffic through a SOCKS5 proxy server, whereas you can establish a virtual encrypted tunnel VPN to a network directly from a DD-WRT powered router. Personally I have a very cheap router TP-Link TL-WR841ND v8, I have never thought that this device could have all these features that are available now thanks to the DD-WRT firmware. I use SOCKS5 proxies regularly and I need to configure browsers, change system settings every time configuration changes. Even more, I haven’t managed to configure system-wide proxy on macOS, some applications don’t respect proxy settings and send network traffic directly. As a result I decided to configure a transparent proxy/redirector to ensure that all traffic is really forwarded to a proxy server and there is no better place other than a router to control all network communications.

This tutorial will guide you through the process of configuring the network-wide proxy redirector using Redsocks and a router with DD-WRT installed.

Prerequisites

In order to complete this tutorial you need basic networking and administration knowledge. Nevertheless I will try to explain each step as detailed as possible. Furthermore you will need the following software/hardware:

• A DD-WRT powered router (with a DD-WRT firmware installed)
• A real machine or virtual machine on your network to act as a transparent proxy redirector

What is DD-WRT ?

As I have already mentioned DD-WRT is a firmware that boost up your router revealing a lot of useful features that are not available with the default firmware installed. Under the hood DD-WRT is a Linux-based firmware, yeah it is a tiny Linux OS running in your router. Surely, this is not plain Linux like you may have running on your desktop, it is modified to satisfy router requirements.

One of the most important features is that DD-WRT comes with a nice, intuitive WEB UI. I enjoy working in the terminal, but you should have a really good memory to remember (because when you are configuring a router you may not have the Internet access) all configuration options and commands that are used to configure a router. However, if despite that you still want to use command line to configure your router, try OpenWRT.

I won’t cover the process of installing the DD-WRT firmware on a router, as far as there is a variety of different routers that support DD-WRT and it is impossible to cover this even in several articles. Hence, if your router supports DD-WRT find a tutorial describing the process of installing DD-WRT on your router. In most cases the steps don’t differ from the official firmware update process.

What is a transparent proxy/redirector ?

A transparent proxy – is a server that receives your request and then fetches requested resource, gets the responses and returns the result to you, so this server sits between you and the outer world. The main feature of a transparent proxy is that it doesn’t modify your requests and just sends them to other servers. Mostly these proxies are used to cache requests and usually a client is not aware of using proxy, thus this type of proxy server is called transparent. Here are some common uses of a transparent proxy:

• Caching Proxy – caching responses in order to accelerate web browsing and reduce the response time.
• Filtering traffic – Forbid or allow certain domains to be accessed by a client.
• Gateways – Acting as a gateway from a local network or a corporate network.

A transparent redirector – is an application that just directly forwards all your packets to a proxy server. It differs from a transparent proxy in not fetching a requested resource, but instead it simply redirects a complete request to a proxy server. Consequently, in it’s turn a proxy server fetches the requested resource for you and therefore your real IP is concealed on a proxy server’s side, not by a transparent redirector like Redsocks.

Transparent redirectors frequently used as a system-wide proxy, all packets in the system are forwarded to a process running locally (or it can be running on the other machine) and a redirector process sends all received packets to a proxy server according the configuration file. This is like a postman or a delivery company carries packets from sender to receiver without modifying the content of a packet (at best).

The Network topology

In the figure above you can see the network topology we are going to build. The topology is quite simple, however it is crucial to understand how does a network packet flows through the network.

Let’s follow the packet flow through the network step by step.

1. A client in the network sends a request, for example to access https://google.com.ua. The network packet is send from the client’s OS to our router, since the default gateway is set to the IP address of the router.
2. The router receives the packet and than checks where to route the packet according rules defined in it’s memory (NVRAM). We are going to configure iptables rules later in this article. For just this case consider that the client matches defined rules and therefore the packet is redirected to the proxifying machine.
3. So the packet is now being redirected to the proxifying machine. When the proxifier(Redsocks) receives the packet it just forwards/redirects it to the defined proxy servers. On this stage the packet may be wrapped according to a proxy protocol, for instance the authentication headers may be added. And after that the packet is sent back to the router.
4. The router forwards this packet out of the local network to the Internet.
5. On this phase the packet travels through different routers, ASes, through your ISP and further in order to reach the destination proxy server (in our case it is a SOCKS5 proxy server).
6. When the packet reaches the destination proxy server. It is handled by the proxy server. The proxy server fetches the requested resource, in this case the main google search page and afterwards returns the result back to you.
7. Now the packet completes the reverse path through Internet, ISP to the router.
8. The router receives the incoming response and redirects it to the origin, the proxifying machine.
9. The proxifier in it’s turn sends the response back to the requesting party, the client in our network, but firstly the packet gets to the router, as far as only router knows all paths in the network and keeps NAT and routing tables.
10. Finally, the client gets the response, in this case the google page will be rendered in a browser.

The packet path is relatively long, to reduce the response time, we could implement redirection to a SOCKS5 proxy server directly on the router, but unfortunately my router is not powerful enough, to handle this by itself, it won’t even establish an OpenVPN connection. If you have a more powerful router, you could try to compile appropriate software and configure it to redirect packets directly to a proxy server.

Step 1 — Flashing DD-WRT firmware on the router

You can find out whether your router supports the DD-WRT firmware on the Supported Device page.

Find and follow the instructions for flashing the firmware on your router. After flashing the firmware you should be able to access the DD-WRT Web UI. In my case it looks as follows

Enabling SSH Management

As far as we will execute bash commands, we need a way to enter these commands. You can use the WEB UI to execute commands, but I prefer doing this from the terminal. In order to configure the Remote SSH Management for your router, first of all access the Web Management UI, in the event of my local network it is accessible at 192.168.0.1.

First of all, navigate to the Service tab. Then, scroll down to the Secure Shell section.

In this section enable SSHd. Next, set the port number for accessing the router via SSH, by default it is 22, so if you have port 22 already forwarded, you have to choose another port. After, you can allow using password, however, it would be better to use SSH keys for security reasons. So if you will be using SSH keys, disable Password Login and paste your public key in the Authorized Keys text area. It should be formatted in the following way.

Default credentials

If you decided to use the Password Authentication, then the username is root and the password is admin by default.

Save new settings by clicking on the Apply Settings button. Next go to the Administration tab and find the Remote Access section.

Enter the same port number as you have entered in the previous tab and enable the SSH Management option. Then, save settings.

Now you should be able to connect to your router using a similar command, don’t forget to setup you SSH keys correctly.

ssh -p [email protected]

In case you are a Windows user, you need to use Putty for connecting to your router via SSH. Open Putty and enter the router’s IP address and the port number you have just set.

If you are using SSH keys for authentication, then go to the SSH settings category and then to the Auth category. Select your private key, it should be in the PPK Format. You can use Puttygen to convert SSH keys.

Don’t forget to save the session settings. Now try to open a connection to the router (use root for username), you might be asked to enter a passphrase if your private key is protected.

Step 2 — Setting up a proxifying machine

Now, we need to configure a machine that will have Redsocks running and as a result act as a “proxifier”. You are free to use a real machine, but usually it’s irrationally to use a separate machine just for this purpose. Therefore, I will use a virtual machine. Another great choice would be an embedded computer like Raspberry PI, Onion Omega, Orange Pi, etc. But I would suggest not to use WiFi connection in case of embedded device, but use the Ethernet interface instead.

I have a PC at home with Proxmox installed, so I will create a virtual machine for this purpose and install Ubuntu 16.04 distro there.

Next, before continue we need to ensure that our machine will always have the same IP address assigned. The guy that is responsible for that is called DHCP server and it is running on your router. We need to configure a static DHCP lease to assign a static IP address to a specific machine.

First of fall you need to find out the mac address of your machine. In my case I can find it through the Proxmox WEB UI in the configuration of the virtual machine. After you have the mac address of your machine. Open the DD-WRT UI and navigate to the Services (the Services tab) and find the “Static Leases” table on the page. Click Add and fill in all required fields. In my case I have the following data entered.

Click Apply Settings, you may need to clear the lease cache and request a new IP address from the DHCP server. Ensure that to the correct IP address was assigned to your machine.

Step 3 — Compiling and installing Redsocks

Next we need to install Redsocks, a transparent redirector. I have written a complete article about installing and configuring Redsocks. Please follow the instructions, but do not set any iptables rules for now or you may set some rules just to check that Redsocks is working properly.

Please note, that you should set the local_ip property to 0.0.0.0 to be able to connect externally. Here is my configuration file (/etc/redsocks.conf).

base { log_info = on; daemon = on; log_debug = on; log = “file:/var/log/redsocks.log”; redirector = iptables; } redsocks { local_port = 12345; ip = 33.33.33.33; local_ip = 0.0.0.0; disclose_src = false; type = socks5; port = 33333; }

One more important point here is that packets will be sent to different ports on the proxifying machine depending on a protocol used, however the Redsocks process listens to one specific port number. Therefore, we need to redirect all incoming packets to the port the Redoscks process listens to. You can achieve this with the following rule.

iptables -A PREROUTING -t nat -i ens18 -p tcp -m multiport –dports 80,443 -j REDIRECT –to-port 12345

Step 4 — Adding Iptables rules on the router

Now we have reached the most important part of the tutorial, we are going to set iptables routing rules in our DD-WRT powered router.

If credentials are valid, you should see greeting from your router.

You may have different cases and goals, so apply your iptables rules instead. In this article I am going to redirect all HTTP/HTTPS packets from a specific host to the transparent redirector.

First of all we need to know what packets should be redirected through the proxy in our network. We have two available choices to use NAT or the Mangle Table (MARK). Using the NAT method, due to change of destination and source addresses, an IP packet will be modified. This leads to undesirable consequences, some of them are described here. Accordingly, we will stick to the Mangle table option, since in this case packets remain unmodified.

Under the hood

The MANGLE table is used to modify packets, mostly it used to set the Type of Service (TOS) or Time to Live (TTL) fields in the IP header. Another case when it comes in handy is to MARK packets to somehow group related packets, making it easier to process these packets further. For instance make routing decisions, as you will see later in this article. One important thing in understanding the MARK target is that a packet itself is not modified at all, all information about marks is stored in the Kernel memory. If you are lucky (you have specific kernel modules loaded) you may find information about individual packets and connections in the pseudo /proc/ filesystem in /proc/net/ip_conntrack and /proc/net/nf_conntrack respectively.

Let’s set rules to mark only appropriate packets (in my case HTTP/HTTPS packets).

iptables -I PREROUTING 1 -t mangle -s 192.168.0.113 ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -p tcp -m multiport –dports 80,443 -j MARK –set-mark 3

I think this rule requires a brief explanation.

• -I PREROUTING 1, Insert (not append -A) the rule into a specific position of the PREROUTING chain. I am inserting the rule and not appending (-A) it on account of the priority of rules. At first by appending new rules at the end of the ruleset I got undesired behavior, my rule was overridden by the rule above.
• -t mangle, Inserting the rule into the mangle table. Please note that this table is present not only in the PREROUTING chain.
• -s 192.168.0.113, This rule matches only if the source IP address is equal to the defined value (192.168.0.113 in this case).
• ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask`, This is the most interesting part. We need to exclude local addresses from being proxified. Note the exclamation sign. The commands nvram get lan_ipaddr and nvram get lan_netmask are just simple bash commands that are available on DD-WRT flashed routers because of special tool /usr/sbin/nvram. This tool is primary used for change/retrieve the content (configuration) stored in the persistent storage.
• -p tcp -m multiport –dports 80,443, As far as both of HTTP and HTTPS protocols are based on the TCP protocol we are setting protocol to tcp and ports used accordingly.
• -j MARK –set-mark 3, Finally, we are jumping to (-J) the MARK target and setting a packet’s mark to 3.

The next rule is similar to the previous one, but we are jumping to the different target CONNMARK. This target is used to mark a whole connection, whereas the MARK target marks individual packets.

iptables -I PREROUTING 2 -t mangle -s 192.168.0.113 ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -p tcp -m multiport –dports 80,443 -j CONNMARK –save-mark

The –save-mark target is used to apply a packet mark onto a connection, resulting in the whole connection being marked according the packet.

In addition we need to exclude our public IP from routing through the proxy server. I have faced with the interesting problem. After applying rules above I wouldn’t be able to access local network resources that are available publicly and resolve to my public IP, like someresource.crosp.net. To get the public IP address on a DD-WRT router we can use the following command.

nvram get wan_ipaddr

As a consequence, we will have the following rules for iptables.

iptables -I PREROUTING 1 -t mangle -s 192.168.0.113 ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -p tcp -m multiport –dports 80,443 -j MARK –set-mark 3 iptables -I PREROUTING 2 -t mangle -s 192.168.0.113 ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -p tcp -m multiport –dports 80,443 -j CONNMARK –save-mark iptables -I PREROUTING 3 -t mangle -s 192.168.0.113 ! -d `nvram get wan_ipaddr` -p tcp -m multiport –dports 80,443 -j MARK –set-mark 3 iptables -I PREROUTING 4 -t mangle -s 192.168.0.113 ! -d `nvram get wan_ipaddr` -p tcp -m multiport –dports 80,443 -j CONNMARK –save-mark

The next and last step is to add static routing rules.

ip rule add fwmark 3 table 13 ip route add default via 192.168.0.145 table 13

We are redirecting all packets marked with the number 3 into the table numbered 13, and adding the default route through the proxifing machine to the table. 192.168.0.145 is the IP address of the proxifying machine.

All in all, here are all rules or the bash script we need to apply to get packets redirected to the proxifying machine.

iptables -I PREROUTING 1 -t mangle -s 192.168.0.113 ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -p tcp -m multiport –dports 80,443 -j MARK –set-mark 3 iptables -I PREROUTING 2 -t mangle -s 192.168.0.113 ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -p tcp -m multiport –dports 80,443 -j CONNMARK –save-mark iptables -I PREROUTING 3 -t mangle -s 192.168.0.113 ! -d `nvram get wan_ipaddr` -p tcp -m multiport –dports 80,443 -j MARK –set-mark 3 iptables -I PREROUTING 4 -t mangle -s 192.168.0.113 ! -d `nvram get wan_ipaddr` -p tcp -m multiport –dports 80,443 -j CONNMARK –save-mark ip rule add fwmark 3 table 13 ip route add default via 192.168.0.145 table 13

Important note

All the rules you set will be lost after router reboot. Under these circumstances, if you need to persist rules there different choices you have. For instance you can do this from the WEB UI, navigating to Administration -> Commands and clicking Save Firewall after you have entered your rules.

When you need to disable proxification, you can use the following rules, or simply reboot a router :).

#!/bin/sh PROXIFYING_MACHINE=192.168.0.145 MACHINE_TO_PROXIFY=192.168.0.113 iptables -D PREROUTING -t mangle -s $MACHINE_TO_PROXIFY ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -p tcp -m multiport –dports 80,443 -j MARK –set-mark 3 iptables -D PREROUTING -t mangle -s $MACHINE_TO_PROXIFY ! -d `nvram get lan_ipaddr`/`nvram get lan_netmask` -p tcp -m multiport –dports 80,443 -j CONNMARK –save-mark iptables -D PREROUTING -t mangle -s $MACHINE_TO_PROXIFY ! -d `nvram get wan_ipaddr` -p tcp -m multiport –dports 80,443 -j MARK –set-mark 3 iptables -D PREROUTING -t mangle -s $MACHINE_TO_PROXIFY ! -d `nvram get wan_ipaddr` -p tcp -m multiport –dports 80,443 -j CONNMARK –save-mark while ip rule delete from 0/0 to 0/0 table 13 2>/dev/null; do true; done ip route flush table 13

Step 5 — Putting all pieces together, proxy testing

It is the time to test our configuration.

First of all, run Redsocks on the proxifying machine, ensure that the process listens to the 0.0.0.0, not the localhost (127.0.0.1).

root@proxyfier:/home/crosp# /opt/redsocks/redsocks -c /etc/redsocks.conf root@proxyfier:/home/crosp# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 977/sshd tcp 0 0 0.0.0.0:12345 0.0.0.0:* LISTEN 1174/redsocks tcp6 0 0 :::22 :::* LISTEN 977/sshd

Next set your iptables and routing rules on your router.

Finally, check whether your traffic is proxified now, for example by executing this command on your machine that you have set in the rules.

MacBookCROSP:~ crosp$ curl ‘https://api.ipify.org?format=json’ {“ip”:”22.22.22.22″}

In case of successful configuration you should see the IP address of a proxy server you have used while setting Redsocks.

Step 6 — Making rules/scripts permanent (Optional)

If you need to persist the proxy scripts and do not want to reconfigure all rules anytime the router reboots, then you have multiple choices to achieve this. Here is an article describing all possible ways to create startup scripts. However, in my case I have no option to enable JFFS on the router, since it has restricted hardware resources. There are some advanced options and workarounds to enable JFFS on routers with poor hardware, however this requires some time and effort to get it working.

Therefore, I need to use other methods, like NVRAM to save startup scripts. There is a great number of different routers, each has a different hardware capabilities, consequently I am not able to describe all methods. Try to find a suitable solution for your router or a similar one.

Conclusion

In this article I’ve described how to configure network traffic proxification using a DD-WRT router with Redsocks. There are some important benefits of this approach. First of all, you can transparently use proxy for a single machine, multiple machines or even a whole local network. Secondly, using the mangle table prevents some problems, furthermore packets are not modified and NAT is not used. You can try to create more complex topologies and in case you have more powerful router, you can try to move some software directly on your router doing that will speed up network connections dramatically.

I hope this tutorial was useful for you. If you have any troubles with setting up this configuration, please feel free to leave comments below.

Enable Script Disable Script

The post Routing network traffic through a transparent SOCKS5 proxy using DD-WRT appeared first on Oleksandr Molochko.

]]>

Redsocks is the tool that allows you to proxify(redirect) network traffic through a SOCKS4, SOCKS5 or HTTPs proxy server. It works on the lowest level, the kernel level (iptables). The other possible way is to use application level proxy, when the proxy client is implemented in the same language as an application is written in. Redsocks operates on the lowest system level, that’s why all running application don’t even have an idea that network traffic is sent through a proxy server, as a result it is called a transparent proxy redirector.

Prerequisites

If you are reading this article, you should probably have an idea why do you need to use a proxy server. Furthermore, you should be acquainted with basic proxy terms and definitions in order to understand everything described in this tutorial. It wouldn’t hurt to have some Linux administration skills as well.

For this tutorial I will be using Centos 7 Minimal installation, it has only the most essential applications installed. But you are free to use any distro, you may skip some steps then.

How does Redsocks work ?

Before start installing Redsocks, I think it is always worth to know how something works internally. And it will help to understand this tool better and therefore troubleshoot issues.

First of all, Redsocks leverages features provided by the Linux kernel firewall (Netfilter module)

I hope this diagram will help you to understand the flow of a packet while using Redsocks.

Here is a brief explanation how does a packet get redirected to Redsocks.

1. When a packet is sent from an app in the system it is handled by the kernel, it triggers the NT_ hooks (defined in <linux/netfilter.h>).
2. The ip_tables kernel module registers at netfilter’s hooks and proceed them according to defined rules.
3. We are defining rules using the iptables command line utility (these rules will be described later in this article). And they are defined in the way that some or mostly all packets are redirected to the local process listening on a port, 12345 in case of Redsocks.
4. All redirected packets are handled by Redsokcs conforming to the configuration file rules.
5. Redsocks redirects packets to proxy servers.

Step 1 — Getting Redsocks source code

First and foremost, you need to update repositories and installed software on the system.

In order to install Redsocks we need to compile it at first. Change into a directory you want to keep source code in.

By default the git command line tool is not installed in most Centos distros. So install it at first.

And clone Redsocks source code from the git repository.

Now change the directory to the redsocks.

Try to compile the application, but it will probably fail. In my case I don’t have even make installed.

Step 2 — Compiling Redsocks

Firstly you need to install build-tools(Development Tools) if you haven’t already.

Next, we need to install dependencies to successfully compile Redsocks.

After installation try to compile Redsocks again using the make command. If compilation succeed you should see the compiled binary file in the current directory.

Now you can copy the binary file to any folder defined in the $PATH variable, to be able to execute it without specifying a full path to Redsocks.

Step 3 — Setting Iptables Rules

To redirect necessary packets to Redsocks we need to define some iptables rules. I will use the rules suggested on the official Redsocks page.

The rule above creates a new custom chain in the NAT table.

Next we need to exclude all local and reserved network addresses. As a result all packets with the destination address from the following ranges will not be sent to Redsocks.

Now we need to add a rule that will redirect all packets from our custom REDSOCKS chain to the local port, we will use the default one – 12345

Please note, this rule doesn’t redirect every packet sent in the system to the port 12345, it only redirects packets that are already passed inside the REDSOCKS chain. You can check this, for example using the following command.

Consequently, we need to define a rule that will redirect chosen packets by some criteria to the REDSOCKS chain. You are free to apply any rules you want, but I will show how to redirect all HTTP and HTTPS packets through a proxy. Define the following rules.

Also you may need to define the following rules in the PREROUTING chain. For redirecting incomming packets to the REDSOCKS chain.

Do not be confused about these rules. They are applied only for incoming packets, that’s why the chain is called PRE ROUTING. Here is the diagram that shows the packet flow through iptables tables and chains.

Furthermore, you are free to define any rules you need, just remember to jump (-j REDSOCKS) to the REDSOCKS chain. Here another example from the official documentation. It redirects only packets sent from a specific user, this could be very useful in some cases. You can follow the user per application strategy (similar used in Android), as a result you will be able set application specific rules.

One more important point, avoid using the root user in iptables rules or you may get stuck in the infinite loop. Here is a problem I’ve encountered because of my inattention.

Having set iptables rules the final step is to configure Redsocks itself using the configuration file.

Step 4 — Configuring Redsocks

Create a file with the name redsocks.conf in the same directory (or any other directory) where the binary file is located. And let’s start by defining the base section of the configuration file as follows.

I think each parameter is self-explanatory. You can find a complete description of every option in the example of configuration file in the official repository.

Next we need to define proxies. This is done using redsocks sections. Here is an example of a possible proxy configuration.

As you can see from the configuration above, this is the SOCKS5 proxy configuration. Redsocks will listen to the port 12345. We have provided credentials to authenticate ourselves on the proxy server as well.

You create multiple redsocks sections. But you have to specify a different local port and as a result you need to set appropriate iptables rules.

As an example, you can use the tricky technique to simulate load balancing with the help of the random module.

There are lot of other ways to define iptables rules for using with multiple proxies, for instance you can use the mangle table, create another chain (ex. REDSOCKS_HTTP) and so on.

Step 5 — Testing proxies with Redsocks

Now it is time to test our configuration. Save the configuration file. Navigate to the folder with the Redsocks compiled executable file. And execute the following command.

If your configuration file is valid and there are no other errors you should almost return immediately in case or running in the daemon mode or see the similar output with the daemon mode off.

Furthermore, you can use the netstat tool to get list of processes bound to ports.

Now try to make any request as the specified user or to match a rule you defined on your own. For example, use the following curl request to get your external IP address.

If the redsocks process is attached to the terminal, you should see something like that in stderr.

If you encountered any error check debug output in the first place.

Conclusion

In this tutorial I’ve described how to install Redsocks on Centos 7. Redsocks provides a very convenient way to configure a proxy environment, starting from a single proxy server to a dozens of proxies of different type. In addition if you are missing some features in the original project, you can check out a fork – Redsocks2.

I hope this tutorial was useful for you. If you have any troubles with setting up Redsocks, please feel free to leave comments below.

The post How to install and configure Redsocks on Centos Linux appeared first on Oleksandr Molochko.