Skip to content

Commit fce436d

Browse files
oxyjohanoxyjohan
authored
Add files via upload
1 parent 95d3bc7 commit fce436d

10 files changed

+450
-0
lines changed

‎README.md

+191
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,191 @@
1+
# Oxylabs’ Residential Proxies integration with Selenium using Java
2+
3+
[<img src="https://img.shields.io/static/v1?label=&message=Java&color=brightgreen" />](https://github.com/topics/java) [<img src="https://img.shields.io/static/v1?label=&message=Selenium&color=orange" />](https://github.com/topics/selenium) [<img src="https://img.shields.io/static/v1?label=&message=Web-Scraping&color=yellow" />](https://github.com/topics/web-scraping) [<img src="https://img.shields.io/static/v1?label=&message=Rotating%20Proxies&color=blueviolet" />](https://github.com/topics/rotating-proxies)
4+
5+
- [Introduction](#introduction)
6+
- [Prerequisites](#prerequisites)
7+
- [Requirements](#requirements)
8+
- [Running the Code](#running-the-code)
9+
- [Compiling the Source Code](#compiling-the-source-code)
10+
- [Running the Jar](#running-the-jar)
11+
- [Proxy Authentication](#proxy-authentication)
12+
- [Testing Proxy Connection](#testing-proxy-connection)
13+
- [Getting Country Specific Proxy](#getting-country-specific-proxy)
14+
- [SSL Support](#ssl-support)
15+
- [Installation Certificate for macOS](#installation-certificate-for-macos)
16+
- [Installation Certificate for Windows](#installation-certificate-for-windows)
17+
- [Understanding the Code](#understanding-the-code)
18+
19+
## Introduction
20+
21+
Integrating proxies that need authorization using the Selenium framework and Java programming
22+
language can be challenging.
23+
24+
This tutorial contains complete code demonstrating how [Oxylabs’ Residential Proxies](https://oxylabs.io/products/residential-proxy-pool) can be
25+
easily integrated with Selenium using Java.
26+
27+
## Prerequisites
28+
29+
- Download and install [Maven](https://maven.apache.org/download.cgi).
30+
- Download and install [Java SE Development Kit](https://www.oracle.com/java/technologies/downloads/).
31+
- Download and install [Google Chrome](https://www.google.com/chrome).
32+
33+
## Requirements
34+
35+
To make this integration easier, we used [BrowserMob Proxy](https://github.com/lightbody/browsermob-proxy)
36+
as a middle layer. It runs proxies locally in JVM and allows chaining of Oxylabs' authenticated proxies.
37+
If you’re using Maven, add this dependency to the `pom.xml` file:
38+
39+
```xml
40+
41+
<dependency>
42+
<groupId>net.lightbody.bmp</groupId>
43+
<artifactId>browsermob-core</artifactId>
44+
<version>2.1.5</version>
45+
</dependency>
46+
```
47+
48+
The other library used in this project – [WebDriverManager](https://github.com/bonigarcia/webdrivermanager),
49+
is optional. It just makes downloading and setting up [Chrome Driver](https://chromedriver.chromium.org/downloads)
50+
easier. To use this library, include the following dependency in `pom.xml` file:
51+
52+
```xml
53+
54+
<dependency>
55+
<groupId>io.github.bonigarcia</groupId>
56+
<artifactId>webdrivermanager</artifactId>
57+
<version>5.0.2</version>
58+
</dependency>
59+
```
60+
61+
If you don’t want to use WebDriverManager, download the Chrome Driver and set the system property as follows:
62+
63+
```java
64+
System.setProperty("webdriver.chrome.driver","/path/to/chromedriver");
65+
```
66+
67+
## Running the Code
68+
69+
### Compiling the Source Code
70+
71+
This is a maven project. To compile this project, run the following command from terminal:
72+
73+
```shell
74+
mvn clean package
75+
```
76+
77+
This will create the file `oxylabs.io-jar-with-dependencies.jar` in the `target` folder.
78+
79+
### Running the Jar
80+
81+
To run the jar, execute the following command from the terminal:
82+
83+
```shell
84+
java -cp target/oxylabs.io-jar-with-dependencies.jar ProxyDemo
85+
```
86+
87+
## Proxy Authentication
88+
89+
Open [ProxySetup.java](src/main/java/ProxySetup.java) file and update your username, password, and endpoint.
90+
91+
```java
92+
static final String ENDPOINT="pr.oxylabs.io:7777";
93+
static final String USERNAME="yourUsername";
94+
static final String PASSWORD="yourPassword";
95+
```
96+
97+
You shouldn’t include the prefix `customer-` in the `USERNAME.` This will be added in the code for country-specific proxies.
98+
99+
## Testing Proxy Connection
100+
101+
Open this project in IDE, open the [ProxySetup.java](src/main/java/ProxySetup.java) file, and run the `main()` function.
102+
This will print two IP addresses.
103+
104+
- The first IP address will be completely random;
105+
- The second IP address will be a country-specific IP address in Germany.
106+
107+
## Getting Country Specific Proxy
108+
109+
Open [ProxyDemo.java](src/main/java/ProxyDemo.java) file and send a two-letter country code to the function `CountrySpecficIPDemo`.
110+
111+
```java
112+
countrySpecificIPDemo("DE");
113+
```
114+
115+
The value of this parameter is a case-insensitive country code in two-letter [3166-1 alpha-2 format](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2). For example, `DE` for
116+
Germany, `GB` for the United Kingdom, etc. For more details, see Oxylabs’ [documentation](https://developers.oxylabs.io/residential-proxies/?java#select-country).
117+
118+
## SSL Support
119+
120+
This code uses BrowserMob Proxy, which supports full MITM.
121+
122+
You may still see invalid certificates warnings. To solve this, install the [ca-certificate-rsa.cer](certificates/ca-certificate-rsa.cer) file in your browser or HTTP client. Alternatively you can generate your own private key rather than using the .cer files distributed with repository.
123+
124+
### Installation Certificate for macOS
125+
- Open Keychain Access
126+
- Selct `System` on left bar
127+
- Click the padlock icon next to `System` and enter your password when prompted
128+
- Select `Certificates` tab on the top
129+
- Drag and drop the `ca-certificate-rsa.cer` file into the `Certificates` tab. You will notice a new certificate appears in the `Certificates` tab with the name `LittleProxy MITM`.
130+
131+
![](images/Keychain.png)
132+
- Right click the certificate and select `Get Info`
133+
- Select `Always Trust`, close the dialog, and enter password again when promoted
134+
135+
![](images/mcos_trust.png)
136+
137+
138+
139+
### Installation Certificate for Windows
140+
141+
- Open the `ca-certificate-rsa.cer` file in Windows Explorer.
142+
- Right click the file, and select `Install`.
143+
- In the Certificate Import Wizard window, click `Browse` and select `Trusted Publishers`. Click `OK` to continue.
144+
145+
![](images/Windows_Step1.png)
146+
- If you see a Security Warning, select `Yes`.
147+
148+
![](images/Windows_Step2.png)
149+
- Follow the wizard to complete the installation.
150+
151+
## Understanding the Code
152+
153+
All the complexity of setting up BrowserMob Proxy and Chrome Options is hidden in
154+
the [ProxyHelper](src/main/java/ProxyHelper.java) class.
155+
156+
In most cases, you should be able to use this file directly without any change.
157+
158+
To create a Chrome Driver instance, go through a two-step process as follows:
159+
160+
First, create an instance of `BrowserMobProxyServer`. This is where you need to provide the proxy endpoint, username, and password.
161+
162+
The fourth parameter is a two-letter country code. If you don’t need a country-specific proxy, set it to `null`:
163+
164+
```java
165+
BrowserMobProxyServer proxy=ProxyHelper.getProxy(
166+
ProxySetup.ENDPOINT,
167+
ProxySetup.USERNAME,
168+
ProxySetup.PASSWORD,
169+
countryCode)
170+
```
171+
172+
Next, call the `ProxyHelper.getDriver()` function.
173+
174+
This function takes up two parameters -`BrowserMobProxyServer` and a `boolean` headless. To run the browser in headless mode, send `true`:
175+
176+
```java
177+
WebDriver driver=ProxyHelper.getDriver(proxy,true);
178+
```
179+
180+
`driver` is an instance of Chrome Driver.
181+
182+
Now, you should write your code to use the Chrome Driver.
183+
184+
Before exiting, remember to close the driver and stop the proxy:
185+
186+
```java
187+
driver.quit();
188+
proxy.stop();
189+
```
190+
191+
If you're having any trouble integrating Oxylabs’ Residential Proxies with Selenium and this guide didn't help you – feel free to contact our customer support at [email protected].

‎certificates/ca-certificate-rsa.cer

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
-----BEGIN CERTIFICATE-----
2+
MIIDfzCCAmegAwIBAgIVAMFQpicWi3EjPX08LgeuA8nAOEfIMA0GCSqGSIb3DQEB
3+
DQUAMEYxGTAXBgNVBAMMEExpdHRsZVByb3h5IE1JVE0xKTAnBgNVBAoMIExpdHRs
4+
ZVByb3h5IFJTQSBJbXBlcnNvbmF0aW9uIENBMB4XDTE1MDEwMjAwMDAwMFoXDTI1
5+
MDEwMjAwMDAwMFowRjEZMBcGA1UEAwwQTGl0dGxlUHJveHkgTUlUTTEpMCcGA1UE
6+
CgwgTGl0dGxlUHJveHkgUlNBIEltcGVyc29uYXRpb24gQ0EwggEiMA0GCSqGSIb3
7+
DQEBAQUAA4IBDwAwggEKAoIBAQC141M+lc046DJaNqIARozRPROGt/s5Ng1UOE84
8+
tKhd+M/REaOeNovW+42uMa4ZifJAK7Csc0dx54Iq35LXy0tMw6ly/MB0pFi+aFCJ
9+
VzXZhbAWIsUmjU8t6z2Y0sjKVX/g3HkdXqaX94jlDtsTjeQXvFhiJNRlX/Locc/f
10+
/oNYZWhg7IPGyQglRY9Dco9kZMSbh5y0yfM8002PNPbNOP4dMX4yYqovT90XbvQ2
11+
rCBbiS6Cys7j44vwOcra9srlb3YQiOCOsYCf7eIhT1GH8tqQ84CHblufqxcGIvXv
12+
V1ex6bDFy63tiPySsOwuVnZglkQ0MDl1GMKVySdPw/qQM5v9AgMBAAGjZDBiMB0G
13+
A1UdDgQWBBRFMQtpkCyZIK9NxaEJDvbfaV1QOzAPBgNVHRMBAf8EBTADAQH/MAsG
14+
A1UdDwQEAwIBtjAjBgNVHSUEHDAaBggrBgEFBQcDAQYIKwYBBQUHAwIGBFUdJQAw
15+
DQYJKoZIhvcNAQENBQADggEBAJuYv1NuxPHom579iAjs19YrFGewHpv4aZC7aWTt
16+
oC1y9418w7QzVOAz2VzluURazUdg/HS9s8abJ8IS0iD0xLz0B1cvJ6F2BezjAwyG
17+
2LxZggmBdLqwjdRkX0Mx3a2HqUpEqaNeKyE8VmzwPuDHN1AqbFcuOPHN7fm7kAtL
18+
4bxFmjgSt7PjEdYwysdjkLC6m+236tuFydpVkXMjuBthsk/hZ1Y/3tbCj/B9a9//
19+
5O+HhYEy+Oa64iFvxfgDfKKUQR3VmwThj1Dh2iJw/kbPJEuQ/PtfcnQhOqyliwg6
20+
Edxd1kaO4HU8Am6TwpmpPFWHRqhM2xj2PAGyfFtN1WfBEQ4=
21+
-----END CERTIFICATE-----

‎images/Keychain.png

258 KB
Loading

‎images/Windows_Step1.png

19.9 KB
Loading

‎images/Windows_Step2.png

12.5 KB
Loading

‎images/mcos_trust.png

334 KB
Loading

‎pom.xml

+106
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://maven.apache.org/POM/4.0.0"
3+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
4+
<modelVersion>4.0.0</modelVersion>
5+
6+
<groupId>io.oxylabs</groupId>
7+
<artifactId>selenium-proxy</artifactId>
8+
<version>1.0</version>
9+
10+
<name>selenium-proxy</name>
11+
<dependencies>
12+
<dependency>
13+
<groupId>org.seleniumhq.selenium</groupId>
14+
<artifactId>selenium-api</artifactId>
15+
<version>4.0.0</version>
16+
</dependency>
17+
<dependency>
18+
<groupId>org.seleniumhq.selenium</groupId>
19+
<artifactId>selenium-chrome-driver</artifactId>
20+
<version>4.0.0</version>
21+
</dependency>
22+
<dependency>
23+
<groupId>net.lightbody.bmp</groupId>
24+
<artifactId>browsermob-core</artifactId>
25+
<version>2.1.5</version>
26+
</dependency>
27+
<dependency>
28+
<groupId>io.github.bonigarcia</groupId>
29+
<artifactId>webdrivermanager</artifactId>
30+
<version>5.0.2</version>
31+
</dependency>
32+
<dependency>
33+
<groupId>org.slf4j</groupId>
34+
<artifactId>slf4j-simple</artifactId>
35+
<version>1.7.32</version>
36+
</dependency>
37+
</dependencies>
38+
39+
<properties>
40+
<maven.compiler.source>17</maven.compiler.source>
41+
<maven.compiler.target>17</maven.compiler.target>
42+
</properties>
43+
<build>
44+
<finalName>oxylabs.io</finalName>
45+
<plugins>
46+
<plugin>
47+
<artifactId>maven-clean-plugin</artifactId>
48+
<version>3.1.0</version>
49+
</plugin>
50+
<plugin>
51+
<artifactId>maven-resources-plugin</artifactId>
52+
<version>3.0.2</version>
53+
</plugin>
54+
<plugin>
55+
<artifactId>maven-compiler-plugin</artifactId>
56+
<version>3.8.0</version>
57+
</plugin>
58+
<plugin>
59+
<artifactId>maven-surefire-plugin</artifactId>
60+
<version>2.22.1</version>
61+
</plugin>
62+
<plugin>
63+
<artifactId>maven-jar-plugin</artifactId>
64+
<version>3.0.2</version>
65+
</plugin>
66+
<plugin>
67+
<artifactId>maven-install-plugin</artifactId>
68+
<version>2.5.2</version>
69+
</plugin>
70+
<plugin>
71+
<artifactId>maven-deploy-plugin</artifactId>
72+
<version>2.8.2</version>
73+
</plugin>
74+
<plugin>
75+
<artifactId>maven-site-plugin</artifactId>
76+
<version>3.7.1</version>
77+
</plugin>
78+
<plugin>
79+
<artifactId>maven-project-info-reports-plugin</artifactId>
80+
<version>3.1.2</version>
81+
</plugin>
82+
<plugin>
83+
<groupId>org.apache.maven.plugins</groupId>
84+
<artifactId>maven-assembly-plugin</artifactId>
85+
<version>3.1.1</version>
86+
<configuration>
87+
<descriptorRefs>
88+
<descriptorRef>jar-with-dependencies</descriptorRef>
89+
</descriptorRefs>
90+
</configuration>
91+
92+
<executions>
93+
<execution>
94+
<id>make-assembly</id>
95+
<phase>package</phase>
96+
<goals>
97+
<goal>single</goal>
98+
</goals>
99+
</execution>
100+
</executions>
101+
102+
</plugin>
103+
104+
</plugins>
105+
</build>
106+
</project>

‎src/main/java/ProxyDemo.java

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
import io.github.bonigarcia.wdm.WebDriverManager;
2+
import net.lightbody.bmp.BrowserMobProxyServer;
3+
import org.openqa.selenium.By;
4+
import org.openqa.selenium.NoSuchElementException;
5+
import org.openqa.selenium.WebDriver;
6+
import org.openqa.selenium.WebElement;
7+
8+
public class ProxyDemo {
9+
10+
public static void main(String[] args) {
11+
randomIPDemo();
12+
countrySpecificIPDemo("DE");
13+
}
14+
15+
private static void countrySpecificIPDemo(String countryCode) {
16+
17+
WebDriverManager.chromedriver().setup();
18+
19+
BrowserMobProxyServer proxy = ProxyHelper.getProxy(
20+
ProxySetup.ENDPOINT,
21+
ProxySetup.USERNAME,
22+
ProxySetup.PASSWORD,
23+
countryCode); // Returns random proxy in that country
24+
WebDriver driver = ProxyHelper.getDriver(proxy, true);
25+
26+
try {
27+
// Selenium Code Here
28+
driver.get("https://ip.oxylabs.io/");
29+
30+
// Parse the response and get IP from <pre>
31+
try {
32+
WebElement preElement = driver.findElement(By.cssSelector("pre"));
33+
System.out.println("Your IP is " + preElement.getText());
34+
} catch (NoSuchElementException e) {
35+
System.out.println("IP Not Found.\n" + driver.getPageSource());
36+
}
37+
38+
} finally {
39+
driver.quit(); // Quit the driver
40+
proxy.stop(); // Quit local JVM proxy
41+
}
42+
43+
}
44+
45+
private static void randomIPDemo() {
46+
WebDriverManager.chromedriver().setup();
47+
48+
BrowserMobProxyServer proxy = ProxyHelper.getProxy(
49+
ProxySetup.ENDPOINT,
50+
ProxySetup.USERNAME,
51+
ProxySetup.PASSWORD,
52+
null); // Return random proxy in any country
53+
WebDriver driver = ProxyHelper.getDriver(proxy, true);
54+
55+
try {
56+
// Selenium Code Here
57+
driver.get("https://ip.oxylabs.io/");
58+
59+
// Parse the response and get IP from <pre>
60+
try {
61+
WebElement preElement = driver.findElement(By.cssSelector("pre"));
62+
System.out.println("Your IP is " + preElement.getText());
63+
} catch (NoSuchElementException e) {
64+
System.out.println("IP Not Found.\n" + driver.getPageSource());
65+
}
66+
67+
} finally {
68+
driver.quit(); // Quit the driver
69+
proxy.stop(); // Quit local JVM proxy
70+
}
71+
}
72+
73+
}

0 commit comments

Comments
 (0)