Comparision of NoSQL databases mongoDB and apache hadoop

NoSQL

Not only SQL (NoSQL) databases using different kind of technics for storing and retrieving data compared to classical relational databases like DB2 or Oracle. They emphasis more on scaling and storing different kind of data then on ACID or CAP support.

Type of NoSQL databases

NoSQL databases can be divided into five different    types:

  • Column
    • Tuple of three values: Unique name, value and timestamp
  • Document
    • semi structured data in documents
  • Key Value
    • dictionary of key value pairs
  • Graph
    • graph oriented system of nodes with properties and edges
  • Multi Model
    • mixture of data models with one backend

MongoDB belongs to the document based NoSQL databases.

HBase as part of apache Hadoop belongs to the column based NoSQL databases.

MongoDB

MongoDB is a open source document database mainly written in C++ as  native application.  The MongoDB Inc. is the company behind that product and offers commercial versions, suppport, etc.

License is A GPL 3.0

Features

  • Document oriented Database
  • aggregation
  • Sharding
  • Replication
  • Indexing
  • automatic fail-over

Apache Hadoop

The Apache Hadoop is a umbrella for different software projects to store and process distributed large data sets. It’s open source and maintained by the apache community.

License is Apache 2.0 license

HDFS

The Hadoop distributed file system (HDFS) is a distributed storage system written in java suitable for storing large files in  a scalable and fault tolerance cluster.

HBase

HBase is a distributed, non relational database modeled after Google BigTable implementation.

Facebook used it for their messages to store over 100 PB on data.

Features

  • Linear and modular scalability.
  • Strictly consistent reads and writes.
  • Automatic and configurable sharding of tables
  • Automatic failover support between RegionServers.
  • Convenient base classes for backing Hadoop MapReduce jobs with Apache HBase tables.
  • Easy to use Java API for client access.
  • Block cache and Bloom Filters for real-time queries.
  • Query predicate push down via server side Filters
  • Thrift gateway and a REST-ful Web service that supports XML, Protobuf, and binary data encoding options
  • Extensible jruby-based (JIRB) shell
  • Support for exporting metrics via the Hadoop metrics subsystem to files or Ganglia; or via JM

Hive

data warehouse extensions for HBase to use HiveQL as like like query language

Ambari

Web based tool to monitor Hadoop installations

Pig

Ability to create MapReduce programs with the PigLatin language to analyze large datasets

Chukwa

Monitoring solution for distributed systems

Zoopkeper

Distributed configuration system

API

For MongoDB exists

For Hadoop exists

  • HBase
    • CLI
    • Java
    • REST
  •  Hive
    • CLI
    • REST

Integrate Hadoop into mongodb

There exists an adapter to use hadoops map reduce functions for aggregations on data. Hadoop jobs extract the data from mongodb, aggregate them and write back to mongodb.

decision criteria to choose the right tool

  • MongoDB
    • best suited as operation database
    • not for data analysis or data processing
  •  Hadoop
    • has many data analysis functionalities
    • use for large amount of read only data

References

Using Vaadin UI with Spring Boot for Spring Data Backend based on MongoDB

With the new Beta 1 of the  Vaadin Addon Vaadin Spring Boot you can use Spring Boot as base framework for your Vaadin UI. The following example application use Spring Data as service layer on top of a mongoDB as NoSQL database. Blackboard is used as generic event sub system inside the Vaadin UI. Complete Source Code is available on github.

Pre Requisites

  • Installed Java 8.x SDK
  • Installed Maven 3.x
  • Running MongoDB 2.x

Source Code

Maven

used maven pom.xml



	4.0.0

	de.schaeftlein.dev.vaadin
	demo-spring-vaadin
	0.0.1-SNAPSHOT
	jar

	demo
	Demo project for Spring Boot with Vaadin

	
		org.springframework.boot
		spring-boot-starter-parent
		1.2.2.RELEASE
		 
	

	
		UTF-8
		demo.DemoApplication
		1.8
		7.4.0
	

	
		
			org.vaadin.addons
			blackboard
			2.2.0
		
		
			com.vaadin
			vaadin-spring-boot
			1.0.0.beta1
		
		
			com.vaadin
			vaadin-themes
			${vaadin.version}
		
		
			com.vaadin
			vaadin-client-compiled
			${vaadin.version}
		
		
			org.springframework.boot
			spring-boot-starter-web
		
		
			org.springframework.boot
			spring-boot-starter-data-mongodb
		
		
			org.springframework.boot
			spring-boot-starter-test
			test
		
	

	
		
			
				org.springframework.boot
				spring-boot-maven-plugin
			
		
	


Command Line Runner

Simple class for starting tomcat as embedded web container for our little application on port 8080

package demo;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class DemoApplication {

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }
}

Simple class for initiate mongodb with some test data

package demo;


import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.CommandLineRunner;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

import demo.data.Customer;
import demo.data.CustomerRepository;

@SpringBootApplication
public class MongoDBApplication implements CommandLineRunner {

	@Autowired
	private CustomerRepository repository;

	public static void main(String[] args) {
		SpringApplication.run(MongoDBApplication.class, args);
	}

	@Override
	public void run(String... args) throws Exception {

		repository.deleteAll();

		// save a couple of customers
		repository.save(new Customer("Alice", "Smith"));
		repository.save(new Customer("Bob", "Smith"));

		// fetch all customers
		System.out.println("Customers found with findAll():");
		System.out.println("-------------------------------");
		for (Customer customer : repository.findAll()) {
			System.out.println(customer);
		}
		System.out.println();

		// fetch an individual customer
		System.out.println("Customer found with findByFirstName('Alice'):");
		System.out.println("--------------------------------");
		System.out.println(repository.findByFirstName("Alice"));

		System.out.println("Customers found with findByLastName('Smith'):");
		System.out.println("--------------------------------");
		for (Customer customer : repository.findByLastName("Smith")) {
			System.out.println(customer);
		}

	}

}

MongoDB Spring Data Repository

Entity class

package demo.data;

import org.springframework.data.annotation.Id;

public class Customer {

	@Id
	private String id;

	private String firstName;
	private String lastName;

	public Customer() {
	}

	public Customer(String firstName, String lastName) {
		this.firstName = firstName;
		this.lastName = lastName;
	}

	@Override
	public String toString() {
		return String.format("Customer[id=%s, firstName='%s', lastName='%s']",
				id, firstName, lastName);
	}

	public String getFirstName() {
		return firstName;
	}

	public void setFirstName(String firstName) {
		this.firstName = firstName;
	}

	public String getLastName() {
		return lastName;
	}

	public void setLastName(String lastName) {
		this.lastName = lastName;
	}

	public String getId() {
		return id;
	}

}

Repository class as service layer

package demo.data;

import java.util.List;

import org.springframework.data.mongodb.repository.MongoRepository;

public interface CustomerRepository extends MongoRepository<Customer, String> {

	public Customer findByFirstName(String firstName);

	public List<Customer> findByLastName(String lastName);

}

Vaadin UI

main application class of the vaadin application

package demo.ui;

import org.springframework.beans.factory.annotation.Autowired;

import com.vaadin.annotations.Theme;
import com.vaadin.navigator.Navigator;
import com.vaadin.server.VaadinRequest;
import com.vaadin.spring.annotation.SpringUI;
import com.vaadin.spring.navigator.SpringViewProvider;
import com.vaadin.ui.Button;
import com.vaadin.ui.CssLayout;
import com.vaadin.ui.Panel;
import com.vaadin.ui.UI;
import com.vaadin.ui.VerticalLayout;
import com.vaadin.ui.themes.ValoTheme;

import demo.ui.event.EventSystem;

@Theme("valo")
@SpringUI
public class MyVaadinUI extends UI {

	@Autowired
	private SpringViewProvider viewProvider;
	
	@Autowired
	EventSystem eventSystem;

	@Override
	protected void init(VaadinRequest request) {
		initLayout();
                registerEvents();
	}

       private void registerEvents() {
		eventSystem.registerEvent(ReloadEntriesEvent.ReloadEntriesListener.class, ReloadEntriesEvent.class);
	}

	private void initLayout() {
		final VerticalLayout root = new VerticalLayout();
		root.setSizeFull();
		root.setMargin(true);
		root.setSpacing(true);
		setContent(root);

		final CssLayout navigationBar = new CssLayout();
		navigationBar.addStyleName(ValoTheme.LAYOUT_COMPONENT_GROUP);
	
		navigationBar.addComponent(createNavigationButton("Default View",
                DefaultView.VIEW_NAME));		
        navigationBar.addComponent(createNavigationButton("MongoDB View",
                MongoDBUIView.VIEW_NAME));
		
		root.addComponent(navigationBar);

		final Panel viewContainer = new Panel();
		viewContainer.setSizeFull();
		root.addComponent(viewContainer);
		root.setExpandRatio(viewContainer, 1.0f);

		Navigator navigator = new Navigator(this, viewContainer);
		navigator.addProvider(viewProvider);
	}

	private Button createNavigationButton(String caption, final String viewName) {
        Button button = new Button(caption);
        button.addStyleName(ValoTheme.BUTTON_SMALL);
        button.addClickListener(event -> getUI().getNavigator().navigateTo(viewName));
        return button;
    }
}

default view as welcome page

package demo.ui;

import javax.annotation.PostConstruct;

import com.vaadin.navigator.View;
import com.vaadin.navigator.ViewChangeListener.ViewChangeEvent;
import com.vaadin.spring.annotation.SpringView;
import com.vaadin.ui.Label;
import com.vaadin.ui.VerticalLayout;

@SpringView(name = DefaultView.VIEW_NAME)
public class DefaultView extends VerticalLayout implements View {
	/*
	 * This view is registered automatically based on the @SpringView annotation. 
	 * As it has an empty string as its view name, it will be shown when navigating to the Homepage
	 */
    public static final String VIEW_NAME = "";

    @PostConstruct
    void init() {
        addComponent(new Label("Welcome View"));
    }

    @Override
    public void enter(ViewChangeEvent event) {
        // the view is constructed in the init() method()
    }
}

mongo db view with a table and a edit form

package demo.ui;

import java.util.List;

import javax.annotation.PostConstruct;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.springframework.beans.factory.annotation.Autowired;

import com.vaadin.event.ItemClickEvent;
import com.vaadin.navigator.View;
import com.vaadin.navigator.ViewChangeListener.ViewChangeEvent;
import com.vaadin.spring.annotation.SpringView;
import com.vaadin.ui.AbstractLayout;
import com.vaadin.ui.Alignment;
import com.vaadin.ui.Button;
import com.vaadin.ui.HorizontalLayout;
import com.vaadin.ui.Table;
import com.vaadin.ui.VerticalLayout;

import demo.data.Customer;
import demo.data.CustomerRepository;
import demo.ui.event.EventSystem;
import demo.ui.event.ReloadEntriesEvent;

@SpringView(name = MongoDBUIView.VIEW_NAME)
public class MongoDBUIView extends VerticalLayout implements View,ReloadEntriesEvent.ReloadEntriesListener{
	public static final String VIEW_NAME = "mongodbui";
	private static final Log LOG = LogFactory.getLog(MongoDBUIView.class);
	
	private Table entityTable;
	private String selectedId;
	private Customer selectedCustomer;
	
	private Button deleteButton;
	private Button editButton;
	
    @Autowired
    private MongoDBContainer mongodbContainer;
    
    @Autowired
    private CustomerForm editForm;
    
    @Autowired
    private CustomerRepository service;
    
    @Autowired
    private EventSystem eventSystem;
	
	@PostConstruct
	void init() {
		registerEvents();
		initData();
		initLayout();
	}

	private void registerEvents() {
		eventSystem.addListener(this);
    }
	
	@SuppressWarnings("serial")
	private void initLayout(){
		setMargin(true);
		setSpacing(true);
		// vaadin table 
        entityTable = new Table();
        entityTable.setContainerDataSource(mongodbContainer);
        entityTable.setVisibleColumns(MongoDBContainer.PROPERTIES);
        entityTable.setColumnHeaders(MongoDBContainer.HEADERS);
        entityTable.setSelectable(true);
        entityTable.setWidth("100%");
        entityTable.setHeight("300px");

        // table select listener
        entityTable.addItemClickListener(new ItemClickEvent.ItemClickListener() {
            @Override
            public void itemClick(ItemClickEvent event) {
                selectedId = (String) event.getItemId();
                selectedCustomer =  mongodbContainer.getItem(selectedId).getBean();

               LOG.info("Selected item id {"+ selectedId+"}");
            }
        });
        // button bar
        final AbstractLayout buttonBar = initButtonBar();
        buttonBar.setWidth("100%");
        
        // edit Form
        editForm.setVisible(false);
        
        addComponent(entityTable);
        addComponent(buttonBar);
        addComponent(editForm);
	}
	
    @SuppressWarnings("serial")
	private AbstractLayout initButtonBar() {
        final HorizontalLayout buttonBar = new HorizontalLayout();

        buttonBar.setSpacing(true);

        final Button addButton = new Button("Add entry");
        editButton = new Button("Edit Entry");
        deleteButton = new Button("Delete entry");

        buttonBar.addComponent(addButton);
        buttonBar.addComponent(editButton);
        buttonBar.addComponent(deleteButton);

        buttonBar.setComponentAlignment(addButton, Alignment.MIDDLE_LEFT);
        buttonBar.setComponentAlignment(editButton, Alignment.MIDDLE_CENTER);
        buttonBar.setComponentAlignment(deleteButton, Alignment.MIDDLE_RIGHT);

        addButton.addClickListener(new Button.ClickListener() {
            @Override
            public void buttonClick(Button.ClickEvent event) {
                editForm.setVisible(true);
            }
        });
        editButton.addClickListener(new Button.ClickListener() {
            @Override
            public void buttonClick(Button.ClickEvent event) {
                editSelectedEntry();
            }
        });
        deleteButton.addClickListener(new Button.ClickListener() {
            @Override
            public void buttonClick(Button.ClickEvent event) {
                removeSelectedEntry();
            }
        });

        return buttonBar;
    }
    
    private void editSelectedEntry() {
        if (selectedId != null) {
        	LOG.info("Edit Customer with id "+selectedId);
            editForm.setData(selectedCustomer);
            editForm.setVisible(true);
        }
    }
    
    private void removeSelectedEntry() {
        if (selectedId != null) {
        	LOG.info("Delete Customer with id "+selectedId);
            service.delete(selectedId);
            eventSystem.fire(new ReloadEntriesEvent());
        }
    }
	
	private void initData() {
		// load data
        List<:Customer> all = service.findAll();
        LOG.info(all);
        // clear table
        mongodbContainer.removeAllItems();
        // set table data
        mongodbContainer.addAll(all);
    }
	
	@Override
	public void enter(ViewChangeEvent event) {
		// the view is constructed in the init() method()
	}

	@Override
	public void reloadEntries(ReloadEntriesEvent event) {
        LOG.info("Received reload event. Refreshing entry table!");
        initData();
        entityTable.markAsDirty();
	}

}

table container definition

package demo.ui;

import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;

import com.vaadin.data.util.BeanContainer;

import demo.data.Customer;


@Component
@Scope("prototype")
public class MongoDBContainer extends BeanContainer<String, Customer> {

	private static final long serialVersionUID = 3090067422968423191L;

	public static final String BEAN_ID = "id";

    public static final Object[] PROPERTIES = {BEAN_ID, "firstName", "lastName"};

    public static final String[] HEADERS = {"ID", "First Name", "Last Name"};

    public MongoDBContainer() {
        super(Customer.class);
        setBeanIdProperty(BEAN_ID);
    }
}

edit form

package demo.ui;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;

import com.vaadin.ui.Button;
import com.vaadin.ui.FormLayout;
import com.vaadin.ui.HorizontalLayout;
import com.vaadin.ui.TextField;

import demo.data.Customer;
import demo.data.CustomerRepository;
import demo.ui.event.EventSystem;
import demo.ui.event.ReloadEntriesEvent;


@Component
@Scope("prototype")
public class CustomerForm extends FormLayout {

    private Logger log = LoggerFactory.getLogger(CustomerForm.class);

    @Autowired
    private CustomerRepository customerService;

    @Autowired
    private EventSystem eventSystem;

    private String id = null;
    private TextField firstName = new TextField("First Name:");
    private TextField lastName = new TextField("Last Name:");

    public CustomerForm() {
        initForm();
    }
    
    public void setData(Customer customer){
    	id = customer.getId();
    	firstName.setValue(customer.getFirstName());
    	lastName.setValue(customer.getLastName());
    }

    private void initForm() {
        addComponent(firstName);
        addComponent(lastName);

        final Button commit = new Button("Commit");
        final Button cancel = new Button("Cancel");

        cancel.addClickListener(new Button.ClickListener() {
            @Override
            public void buttonClick(Button.ClickEvent event) {
                clearAndHide();
            }
        });
        commit.addClickListener(new Button.ClickListener() {
            @Override
            public void buttonClick(Button.ClickEvent event) {
                commitForm();
                fireCommitEvent();
                clearAndHide();
            }
        });

        final HorizontalLayout buttonBar = new HorizontalLayout();

        buttonBar.addComponent(commit);
        buttonBar.addComponent(cancel);

        addComponent(buttonBar);
    }

    private void commitForm() {
        if(id!=null){
        	log.info("Update user with id {}, name {} and address {}", id, firstName.getValue(), lastName.getValue());
        	Customer customer = customerService.findOne(id);
        	customer.setFirstName(firstName.getValue());
        	customer.setLastName(lastName.getValue());
        	customerService.save(customer);
        }
        else {        	
        	log.info("Creating user with name {} and address {}", firstName.getValue(), lastName.getValue());
        	customerService.save(new Customer(firstName.getValue(), lastName.getValue()));
        }
    }

    private void clearAndHide() {
        firstName.setValue("");
        lastName.setValue("");
        id = null;
        setVisible(false);
    }

    private void fireCommitEvent() {
        log.info("Fire commit event!");
        eventSystem.fire(new ReloadEntriesEvent());
    }
} 

Vaadin Event System

Event sub system System

package demo.ui.event;

import java.io.Serializable;

import org.springframework.context.annotation.Scope;
import org.springframework.stereotype.Component;

import com.github.wolfie.blackboard.Blackboard;
import com.github.wolfie.blackboard.Event;
import com.github.wolfie.blackboard.Listener;

@Component
@Scope("session")
public class EventSystem implements Serializable {
	private static final long serialVersionUID = 7829012291289167478L;
	private Blackboard blackboard = new Blackboard();

    public EventSystem() {
        init();
    }

    private void init() {
        blackboard.enableLogging();
    }

    public void registerEvent(Class<? extends Listener> listenerClass, Class<? extends Event> eventClass) {
        blackboard.register(listenerClass, eventClass);
    }

    public <T extends Listener> void addListener(T listener) {
        blackboard.addListener(listener);
    }

    public <T extends Event> void fire(T event) {
        blackboard.fire(event);
    }
}

Custom Reload Event with Listener

package demo.ui.event;

import com.github.wolfie.blackboard.Event;
import com.github.wolfie.blackboard.Listener;
import com.github.wolfie.blackboard.annotation.ListenerMethod;

public class ReloadEntriesEvent implements Event {

	public interface ReloadEntriesListener extends Listener {
		@ListenerMethod
		public void reloadEntries(ReloadEntriesEvent event);
	}

	public ReloadEntriesEvent() {
	}
}

Testing

JUnit Test for Spring Data Repository

package demo;

import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.SpringApplicationConfiguration;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;

import demo.data.Customer;
import demo.data.CustomerRepository;

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = MongoDBApplication.class)
public class MongoDBApplicationTests {
	
	@Autowired
	CustomerRepository repo;
	
	private final static Log LOG = LogFactory.getLog(MongoDBApplicationTests.class);

	@Test
	public void foundAlice(){
		Customer alice = repo.findByFirstName("Alice");
		Assert.assertNotNull(alice);
		LOG.info(alice);
	}
}

Running

Using Command Line Runner

Inside Eclipse run the “demo.DemoApplication” class and open the url http://localhost:8080/ to see the welcome page.

Welcome page

Switch to the mongodb view

mongodb view

Select a row and click on “Edit entry” to change data or “delete entry” to delete selected customer. With “Add entry” you can create a new customer in the mongoDB.

VPN Connection from Ubuntu 13.04 to AVM Fritzbox

Pre Requisites

Install Cisco compatible VPN Client with

sudo apt-get install vpnc network-manager-vpnc network-manager-vpnc-gnome

Fritzbox

  • Login into your Fritzbox as admin
  • Go to System -> Fritzbox Users
  • Click on the edit icon from one of your users
  • Check the VPN Checkbox
  • Click on the Link in the last line for showing the VPN settings
  • Leave the Popup with the settings open

Ubuntu

  • Click on the Connections Menu icon in the toolbar on top
  • Go to VPN Connections -> Configure VPN
  • Click on Add Button at the right side
  • Choose from the list under VPN the Cisco compatible VPN client type
  • Click on create button
  • Enter the following values (<Ubuntu>: <Fritzbox>)
    • Connection Name : Fritzbox
    • Gateway: <Server Adress>
    • User Name: <IPSec Identifier>
    • User Password :<User Password>
    • Group Name: <IPSec Identifier>
    • Group Password: <IPSec Pre-Shared Key>
  • Set the Password Option to “Saved” (optional)
  • Click on save button
  • Cut the cable connection if established
  • Establish a mobile connection
  • Choose from VPN menu the new created VPN connection
  • Try accessing the Fritzbox web interface with your favorite browser

Ad block with Squid3 and Quintolabs instead of Adblock Plus

Motivation

Under the twitter hashtag #adblockgate you find news related to the business of the company behind the adblock plus browser plugin for chrome or firefox.

The problem with their business is the secret white list which serve you ads without a chance to avoid them.

The advantage of this solution here is to have a central ad blocking service for all of your devices including smart phones and tablets.

Used Software (under ubuntu 13.04):

  • Apache Httpd (for serving the pac file)
  • Squid3 (as primary proxy)
  • Quintolabs (as  ICAP Server for ad blocking)

Install Apache Httpd

apt-get install apache2

Install Squid3

apt-get install squid3

Install Quintolabs

open Download page and choose deb File and run the command (as root)

dpkg -i qlproxy-2.0.0.d746b-ubuntu_i386.deb

Create PAC File for the web clients

Change <your ubuntu host ip> with the ip of your server

nano /var/www/proxy.pac


function FindProxyForURL(url, host) {
// avoid anti virus update problems
if (shExpMatch(host,"*.bitdefender.com")) {
return "DIRECT";
}

// avoid anti virus update problems
if (shExpMatch(host,"*.bitdefender.net")) {
return "DIRECT";
}
// avoid proxy for URLs inside your network
if (isInNet(host, "192.168.178.0", "255.255.255.0")) {
return "DIRECT";
}

// the rest goes through your proxy
return "PROXY <your ubuntu host ip>:3128";
}

Configure Squid3

nano /etc/squid3/squid.conf


acl manager proto cache_object
acl localhost src 127.0.0.1/32 ::1
acl to_localhost dst 127.0.0.0/8 0.0.0.0/32 ::1
acl localnet src 192.168.0.0/16 # RFC1918 possible internal network
acl SSL_ports port 443
acl Safe_ports port 80          # http
acl Safe_ports port 21          # ftp
acl Safe_ports port 443         # https
acl Safe_ports port 70          # gopher
acl Safe_ports port 210         # wais
acl Safe_ports port 1025-65535  # unregistered ports
acl Safe_ports port 280         # http-mgmt
acl Safe_ports port 488         # gss-http
acl Safe_ports port 591         # filemaker
acl Safe_ports port 777         # multiling http
acl CONNECT method CONNECT
http_access allow all
http_access allow manager localhost
http_access deny manager
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access allow localhost
http_access deny all
http_port 3128
cache_mem 768 MB
maximum_object_size_in_memory 1024 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
pid_filename /var/run/squid3.pid
buffered_logs on
coredump_dir /var/spool/squid3
refresh_pattern ^ftp:           1440    20%     10080
refresh_pattern ^gopher:        1440    0%      1440
refresh_pattern -i (/cgi-bin/|\?) 0     0%      0
refresh_pattern (Release|Packages(.gz)*)$      0       20%     2880
refresh_pattern .               0       20%     4320
negative_dns_ttl 5 minutes
via off
ignore_expect_100 on
forward_timeout 30 seconds
connect_timeout 30 seconds
read_timeout 30 seconds
request_timeout 30 seconds
persistent_request_timeout 1 minutes
client_lifetime 20 hours
dns_timeout 5 minutes
ipcache_size 10240
forwarded_for delete
client_db off
acl home_network src 192.168.178.0/24
icap_enable on
icap_preview_enable on
icap_preview_size 4096
icap_persistent_connections on
icap_send_client_ip on
icap_send_client_username on
icap_client_username_header X-Client-Username
icap_service qlproxy1 reqmod_precache bypass=0 icap://127.0.0.1:1344/reqmod
icap_service qlproxy2 respmod_precache bypass=0 icap://127.0.0.1:1344/respmod
adaptation_access qlproxy1 allow all
adaptation_access qlproxy2 allow all

Patch Quintolabs

To use own ad block lists change the settings file

nano /opt/quintolabs/qlproxy/bin/settings.py

adblock = {
“DataDir” : macro_var + “/spool/adblock”,
“DataFiles” : {
“easylist.txt” : “https://easylist-downloads.adblockplus.org/easylist.txt”,
“easyprivacy.txt” : “https://easylist-downloads.adblockplus.org/easyprivacy.txt”,
“easylistgermany.txt” : “https://easylist-downloads.adblockplus.org/easylistgermany.txt”,
“fanboy-russian.txt” : “https://secure.fanboy.co.nz/fanboy-russian.txt”,
“own-adblock.txt” : “http://ralf.schaeftlein.com/adblock-plus.txt”
}
}

own-adblock.txt can be used as alias for own ad block subscriptions. See further configuration below.

Change the validation of ad block lists to avoid verification errors with different headers

nano /opt/quintolabs/qlproxy/bin/update_adblock.py

change verify function line to this

if re.search(r’\[Adblock\s+Plus’, line) != None:

Configure Quintolabs

Enable ad block module in the global config

nano /etc/opt/quintolabs/qlproxy/qlproxyd.conf

adblock_enabled = yes

and disable the other modules

urlblock_enabled = no
httpblock_enabled = no
contentblock_enabled = no
adultblock_enabled = no

Enable relaxed module for your subnet

nano /etc/opt/quintolabs/qlproxy/policies/relaxed/members.conf

Change the last line to (assuming 192.168.0.0 is the network subnet of your clients)

user_ip_subnet = 192.168.0.0/16

set pass through domains for your antivirus tool by editing

nano /etc/opt/quintolabs/qlproxy/policies/relaxed/exceptions.conf

Change the lines with domain name for your needs like this for bitdefender

# disable filtering for all third level domains of the following servers
domain_name = .microsoft.com
domain_name = .quintolabs.com
domain_name = .bitdefender.com
domain_name = .bitdefender.net

Enable all ad block subscriptions

nano /etc/opt/quintolabs/qlproxy/policies/relaxed/rules/block_ads.conf

use_subscription = easylist.txt
use_subscription = easylistgermany.txt
use_subscription = fanboy-russian.txt
use_subscription = easylist_custom.txt
use_subscription = easyprivacy.txt
use_subscription = own-adblock.txt

Restart Quintolabs to read change configs

/etc/init.d/qlproxy restart

Update ad block list for the first time

/etc/cron.daily/qlproxy_update

Restart Quintolabs to read change ad block lists

/etc/init.d/qlproxy restart

Restart web and proxy server

restart squid3

service apache2 restart

Configure windows browser

  1. Start internet explorer and open settings.
  2. Go to connection tab
  3. click on lan settings
  4. click second checkbox for “script for automatic configuration” and enter (replace <servername> with your ubuntu hostname)http://<servername>/proxy.pac
  5. Confirm dialog with ok and close settings with ok
  6. Restart all browsers

This affect chrome and internet explorer.

Under Firefox

  1. open settings
  2. go to advanced
  3. go to network
  4. Click on connections
  5. Set the last line to the same url as step 4 from above and click on the radiobox over the url for “automatic proxy configuration url”
  6. Click on “reload” button beside the url text field

Remove existing browser plugins

Uninstall chrome and firefox adblock plus plugins

Set proxy under android for the wlan

  1. Open settings 
  2. Click on wlan (which is enabled)
  3. Hold the entry with your ssid pressed
  4. Choose from menu “change network setttings”
  5. Click on checkbox “show extended settings”
  6. Choose “manual” as proxy settings
  7. Choose <servername> (like in the url above) as proxy hostname
  8. Choose “3128” as proxy port
  9. Click on save button

Test ad block

Open web pages like the following to see if the ad banners are removed by the proxy server

http://www.spiegel.de

http://www.engadget.com/

Check if proxy can be detected by external web sites

Sample sites which looks for http header in the request to see if you are coming via a proxy

http://whatismyipaddress.com/proxy-check

“Proxy server not detected.”

http://www.lagado.com/proxy-test

“This request appears NOT to have come via a proxy.”

Ubuntu 13.04 on Lenovo Thinkpad X60

Hardware Spec

  • 4GB RAM
  • 120 GB Samsung SSD 830 (AHCI Mode)
  • Dual Core L7500 @ 1.6 Ghz

Used Distro

Ubuntu 13.04 x64 Desktop Edition

Installation

Download the iso file and use unetbootin to install on a usb stick. Leave the stick in the laptop. Reboot. Go into bios by pressing ThinkVantage button and then F1. Choose Startup -> Boot If the usb hdd is excluded then include them with x button and move up with F6. Press F10 to save and exit. Power on the laptop and choose “Install ubuntu” to start installation process.

After installation

recommended additional software (install with “apt-get install <name>”)

  • ntp (for correct date and time)
  • xournal (tablet programm for writing combined with stylus notes)
  • libxss1 (prerequisites for google chrome)
  • lingconf2-4 (prerequisites for google chrome)

Google Chrome

Open Download page and click on chrome download button. Choose “64bit .deb” and click on “accept and install” button

Get rid of anoying shopping hints in unity

apt-get remove unity-lens-shopping

Click on the amazon icon in the unity starter with right click and choose remove from starter

Getting special Hardware to work

Hibernate on closing laptop

Test if it works by running

pm-hibernate

If the laptop suspend to disk and start at the current state after reboot everything is fine to go on.

Edit file /etc/polkit-1/localauthority/50-local.d/com.ubuntu.enable-hibernate.pkla

[Re-enable hibernate by default]
Identity=unix-user:*
Action=org.freedesktop.upower.hibernate
ResultActive=yes

Reboot machine and open Energy menu (click on the battery in the top bar and choose energy settings) to define the hibernate state if the laptop cover is closed.

Gnome Buttons on the left side

gsettings set org.gnome.desktop.wm.preferences button-layout ‘:minimize,maximize,close’

Fingerprint Sensor

sudo apt-add-repository ppa:fingerprint/fingerprint-gui && sudo apt-get update

sudo apt-get install libbsapi policykit-1-fingerprint-gui fingerprint-gui

run the fingerprint.gui:

  1. Show Vendor => Choose STMicroelectronics and click next
  2. Choose a finger and click next
  3. Repeat five times to scan and click next
  4. Click on the test button and see if it works

Now you can login or authenticate for sudo with your fingerprint

Source:

 

tech stuff, development news…