IMCODEBASED.COM

Make + Docker makes life easy

2022-10-21T00:00:00+00:00

Summary: Use the power of a makefile to reduce the load of typing big commands for docker interaction through docker cli.

Problem

When I work with docker commands, I don’t remember all available syntax that I am going to use for day to day operations with docker cli, such as create image , run image etc.. .

As an example, let us say I have a defined this nginx docker file and my docker build command will be something like this:

docker build --file $(PWD)/Dockerfile --tag nginx:1.23.1-alpine_$(ENV) --build-arg env=$(ENV) .

Now, every time I want to build my docker image, I will have to run this big docker command.

So why should I waste my time, if I can simply type make build and it will build my docker image.

This is where make will be used.

What is Make?

According to this documentation

Make is a tool which controls the generation of executables and other non-source files of a program from the program’s source files.

Solution

I will create a file called makefile. This file is a collection of targets. Each target contains a collection of commands that it will execute one after the another.

My target will look something like this:

target: 
    @command1
    @command2

In the case of above given docker build example, I will define my target as

build: 
	-@docker build --file $(PWD)/Dockerfile --tag nginx:1.23.1-alpine_$(ENV) --build-arg env=$(ENV) .

I can now call make build and it will execute the above big command for me.

Please have a look at a sample makefile you will find many commands that I have created to work with my nginx docker image.

Now this is the power of using make!

I can now build an image, run, check the status of my nginx container and status of my nginx configuration, with this single command:

make build start status testconfig

Bingo!

You have just replaced 4 big docker commands two 4 words.😆

You can find my nginx sample code here

SQL Queries to IIS Logs

2016-11-15T00:00:00+00:00

Log parser is a powerful, versatile tool that provides universal query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows® operating system such as the Event Log, the Registry, the file system, and Active Directory®.

Overview

There is always a case that you want to integrated IIS Logs into your project. You can relay on third party services/ interface. However, the best approach is to have one application where you can integrate the logic of IIS log reading.

Since 2009, Microsoft has introduced a tool called Log Parser that provides sql like queries against IIS Log files.

You can also download the same through choco command.

Get Started

Go to the path C:\Program Files (x86)\Log Parser 2.2\

Log parser by default provide you a COM DLL that you can import in Native C/C++ projects, or you can also import the same in .NET project that uses Interop facility.

To use COM DLL into .NET project you can also use tlbimp.exe command as well.

Open command prompt and run this simple select statement:

“C:\Program Files (x86)\Log Parser 2.2\LogParser.exe” “select * from c:\inetpub\logs\logfiles\w3svc1\u_ex150101.log”

You can also view output in a GUI through the -o:DataGrid switch value.

”C:\Program Files (x86)\Log Parser 2.2\LogParser.exe” “select * from c:\inetpub\logs\logfiles\w3svc1\u_ex150101.log” -I:w3c -o:datagrid

Command Reference:

logparserplus.com
http://mlichtenberg.wordpress.com/2011/02/03/log-parser-rocks-more-than-50-examples/
LogParser Studio
LogParser.chm file is also a great source of truth 😉

C# Integration

First thing you got to do is Add reference into your .NET project using Visual Studio IDE.

IWebLogService.cs:

public interface IWebLogService
{
    List<IISLogCount> GetLogs(string fileName = null, string api = null);
    List<IISLog> GetLogDetails(string uri, string fileName = null);
}

public class WebLogService : IWebLogService
    {
        public List<IISLogCount> GetLogs(string fileName = null, string api = null)
        {
            if (string.IsNullOrWhiteSpace(fileName))
            {
                fileName = "{0}\\*.log".FormatMessage(ConfigurationManager.AppSettings["IISLOGPATH"]);
            } 

            if (string.IsNullOrWhiteSpace(fileName))
            {
                throw new ArgumentNullException(fileName);
            }

            string query = string.Empty;

            if (string.IsNullOrWhiteSpace(api))
            {
                query = @"
                SELECT date, cs-uri-stem, cs-method, count(cs-uri-stem) as requestcount from {0}
                WHERE STRLEN (cs-username ) > 0 
                GROUP BY date, cs-method, cs-uri-stem 
                ORDER BY date, cs-uri-stem, cs-method, count(cs-uri-stem) desc".FormatMessage(fileName);
            }
            else
            {
                query = @"
            SELECT date, cs-uri-stem, cs-method, count(cs-uri-stem) as requestcount from {0}
                WHERE cs-uri-stem LIKE {1} and STRLEN (cs-username ) > 0 
                GROUP BY date, cs-method, cs-uri-stem 
                ORDER BY date, cs-uri-stem, cs-method, count(cs-uri-stem) desc".FormatMessage(fileName, " '%/api/{0}%' ".FormatMessage(api));
            }

            var recordSet = this.ExecuteQuery(query);
            var records = new List<IISLogCount>();
            int hit = 0;
            for (; !recordSet.atEnd(); recordSet.moveNext())
            {
                var record = recordSet.getRecord().toNativeString(",").Split(new[] { ',' });
                if (int.TryParse(record[3], out hit))
                {                }
                else
                {
                    hit = 0;
                }

                records.Add(new IISLogCount { Hit = hit, Log = new IISLog { EntryTime = Convert.ToDateTime(record[0]), UriQuery = record[1], Method = record[2] } });
            }

            return records;
        }

  public List<IISLog> GetLogDetails(string uri, string fileName = null)
        {
            if (string.IsNullOrWhiteSpace(fileName))
            {
                fileName = "{0}\\*.log".FormatMessage(ConfigurationManager.AppSettings["IISLOGPATH"]);
            }

           if (string.IsNullOrWhiteSpace(fileName))
            {
                throw new ArgumentNullException(fileName);
            }
           string query = string.Empty;

            query = @"SELECT"
            + " TO_TIMESTAMP(date, time) AS EntryTime"
            + ", s-ip AS ServerIpAddress"
            + ", cs-method AS Method"
            + ", cs-uri-stem AS UriStem"
            + ", cs-uri-query AS UriQuery"
            + ", s-port AS Port"
            + ", cs-username AS Username"
            + ", c-ip AS ClientIpAddress"
            + ", cs(User-Agent) AS UserAgent"
            + ", cs(Referer) AS Referrer"
            + ", sc-status AS HttpStatus"
            + ", sc-substatus AS HttpSubstatus"
            + ", sc-win32-status AS Win32Status"
            + ", time-taken AS TimeTaken"
            + " from {0} WHERE cs-uri-stem = '{1}' and STRLEN (cs-username ) > 0  ORDER BY EntryTime".FormatMessage(fileName, uri);

            var resultSet = this.ExecuteQuery(query);

            var records = new List<IISLog>();
            for (; !resultSet.atEnd(); resultSet.moveNext())
            {
                var record = resultSet.getRecord().toNativeString(",").Split(new[] { ',' });

                records.Add(new IISLog { EntryTime = Convert.ToDateTime(record[0]), UriQuery = record[1], Method = record[2], UriStem = record[3], UserAgent = record[6] });
            }

            return records;
        }

        internal ILogRecordset ExecuteQuery(string query)
        {
           LogQueryClass logQuery = new LogQueryClass();
            MSUtil.COMW3CInputContextClass iisLog = new MSUtil.COMW3CInputContextClass();
            return logQuery.Execute(query, iisLog);
        }
    }

This is how my POC looks like:


public class IISLog
    {
        public string LogFilename { get; set; }
        public int RowNumber { get; set; }
        public DateTime EntryTime { get; set; }
        public string SiteName { get; set; }
        public string ServerName { get; set; }
        public string ServerIPAddress { get; set; }
        public string Method { get; set; }
        public string UriStem { get; set; }
        public string UriQuery { get; set; }
        public int Port { get; set; }
        public string Username { get; set; }
        public string ClientIpAddress { get; set; }
        public string HttpVersion { get; set; }
        public string UserAgent { get; set; }
        public string Cookie { get; set; }
        public string Referrer { get; set; }
        public string Hostname { get; set; }
        public int HttpStatus { get; set; }
        public int HttpSubstatus { get; set; }
        public int Win32Status { get; set; }
        public int BytesFromServerToClient { get; set; }
        public int BytesFromClientToServer { get; set; }
        public int TimeTaken { get; set; }
    }

public class IISLogCount
    {
        public IISLog Log
        {
            get;
            set;
        }

        public int Hit { get; set; }
    }

Once the Service class has been defined, you can create your proxy whichever you want it.

Here I am using ApiController driven class as a proxy. This class will be passing any Http request to this service and returning back the response in Http protocol.

   [HttpGet]
    [Route("applications/iislog")]
    public IHttpActionResult GenerateIISLog(string fileName = null, string api = null)
    {
        return Ok(_weblogService.GetLogs(fileName, api));
    }

    [HttpGet]
    [Route("applications/iislogdetails")]
    public IHttpActionResult GenerateIISLogDetails(string uri, string fileName = null)
    {
        return Ok(_weblogService.GetLogDetails(uri, fileName));
    }

I’m using NInject IoC/ DI framework for injecting service to the controller:

kernel.Bind(typeof(IWebLogService)).To(typeof(WebLogService)).InRequestScope();

Points of Interest

As you can see in my above sample that I could managed to read a text file using SQL commands. LogParser is fantastic tool for retrieving usage stats/ bandwidth , slow pages and many more details.

One thing I like the most with LogParser is it can even communicate with Directory Service Logs, Mail Logs, Event View Logs and et al; that too using SQL commands.

Implementing SOLID principle For TDD on Web Service Reference

2016-03-30T00:00:00+00:00

In software engineering, SOLID is a mnemonic acronym for five design principles intended to make object-oriented designs more understandable, flexible, and maintainable.

What is SOLID?

SOLID are five basic principles which help to create good software architecture. SOLID is an acronym where:-

• S stands for SRP (Single responsibility principle
• O stands for OCP (Open closed principle)
• L stands for LSP (Liskov substitution principle)
• I stands for ISP ( Interface segregation principle)
• D stands for DIP ( Dependency inversion principle)

There are many good article present over the Internet explaining SOLID in details. However, my entire focus will be on how I have implemented this principle on Web Service proxy that I’ve to currently worked on.
Since I am following TDD approach, I will start the development from explaining Unit test first:

Unit testing is all about testing current method (High cohesion and Low coupling). If this method is relaying on another method or an object then you should mock that object/ method so that you can test your method independently.

If you not sure about High cohesion and low coupling principle in OOPS, then I recommend you to learn about SOLID Principles.

This principle applies even when the dependent code has not been written as yet (well that is why we call it as TDD ;-).

In case of my integration project with sharepoint I was dependent on an external resource. I’d to call a sharepoint web service method GetUserProfileByName, with the endpoint at http://yourwebsite/_vti_bin/UserProfileService.asmx

When we used Visual Studio, Add Service reference tool, it creates a proxy class to call the web service:


[System.Web.Services.WebServiceBindingAttribute(Name="UserProfileServiceSoap", Namespace="http://microsoft.com/webservices/SharePointPortalServer/UserProfileService")]
 public partial class UserProfileService : System.Web.Services.Protocols.SoapHttpClientProtocol 
{
}

When I saw this proxy file, it has got more than 50+ methods that are exposed. As a principle of SOLID, we should expose only those methods that are important to expose. In our case it is just one method i.e. GetUserProfileByName.
Another important factor we should consider into our design is to how can I assure that the implementation of the sharepoint can be easily replaced with any new external end point such as Active Directory.

To solve all these problems, I just need to follow a I and D pont from SOLID principle.

I – Interface segregation principle – many client-specific interfaces are better than one general-purpose interface.

D – Dependency inversion principle – one should “Depend upon Abstractions. Do not depend upon concretions.”

Now I will not go into the detail of SOLID but these are the steps that you should follow:

Import Web service proxy into the project. Now keep in mind that the web service proxy is of partial class. Thus we can create an another partial class and implement a contract.
Create a Contract that must contain only those members that you need to expose.

public interface IServiceClient
 {
    bool UseDefaultCredentials { get; set; }
    string Url { get; set; }
 }

public interface IUserProfileServiceClient: IServiceClient
 {
 PropertyData[] GetUserProfileByName(string accountName);
 }

Now that I have got contract ready, all I have to do is create a partial class with same signature. In case my case it:

public partial class QueryService : IQueryServiceClient {
 }

NOTE: You don’t need to implement any member because it has been already created by the proxy class.

Create a contract of your implementation that you want to test:

public interface IExternalUserService
 {
 User GetUserProfile(string accountName);
 }

Create a concrete class:

public class DirectoryUserService : IExternalUserService 
{
 private readonly IUserProfileServiceClient _profileService;
public DirectoryUserService(IUserProfileServiceClient profileService) {
_profileService = profileService;
}

public User GetUserProfile(string accountName)
{
var propertyBag = _profileService.GetUserProfileByName(accountName);
}

Now that we have implemented our method that we want to test, the following UnitTest implementation is using Moq Framework for Stub that is going to pass as IUserProfileServiceClient object.


[TestClass]
public class DirectoryUserServiceTest
{
private PropertyData[] _fakePropertyData;

[TestInitialize]
public void Setup()
{
// Arrange
_fakePropertyData = new[]
{ new PropertyData() {
Name = "WorkEmail",
Values = new[] {new ValueData {Value = "amit.x@xxxxx.com.au"}}
},
    new PropertyData()
    {
    Name = "PictureURL",
    Values = new[] {new ValueData {Value = "http://x/xtra/photos/photo152173.jpg"}}
    },
    new PropertyData() {Name = "FirstName", Values = new ValueData[] {new ValueData() {Value = "Amit"}}},
    new PropertyData() {Name = "LastName", Values = new ValueData[] {new ValueData() {Value = "Malhotra"}}},
    new PropertyData() {Name = "DirectoryID", Values = new ValueData[] {new ValueData() {Value = "152173"}}},
    new PropertyData() {Name = "UserProfile_GUID",Values = new ValueData[] {new ValueData() {Value = Guid.NewGuid()}}
    },new PropertyData() {Name = "BusinessUnit", Values = new ValueData[] {new ValueData() {Value = "RBS"}}},

    new PropertyData(){Name = "Department",Values = new ValueData[] {new ValueData() {Value = "Partner Engagement"}}
}
};
}


[TestMethod]
public void TestGetUserProfileNotNull()
{

// Arrange
var mockService = new Mock<IUserProfileServiceClient>();
mockService.Setup(r => r.GetUserProfileByName(It.Is<string>(x => x.Equals("amit"))))
.Returns(_fakePropertyData);

IExternalUserService directoryService = new DirectoryUserService(mockService.Object, null);

// Act
User user = directoryService.GetUserProfile("amit");

//Assert
Assert.IsNotNull(user);
}

Point of Interest:

Understand SOLID:

S stands for SRP (Single responsibility principle):- A class should take care of only one responsibility.
O stands for OCP (Open closed principle):- Extension should be preferred over modification.
L stands for LSP (Liskov substitution principle):- A parent class object should be able to refer child objects seamlessly during runtime polymorphism.
I stands for ISP (Interface segregation principle):- Client should not be forced to use a interface if it does not need it.
D stands for DIP (Dependency inversion principle) :- High level modules should not depend on low level modules but should depend on abstraction.

Understand Moq:

Now if you are following the discipline of TDD and are building large scale enterprise application that interacts with services, databases, or other data sources, writing data driven unit tests becomes a challenge. Test setup (for each test) becomes more complicated and challenging than the actual unit test. This is where the concept of Mock comes into picture