Prashanth Batchu

Reload your Amazon Balance ‘n’ number of times using Scala & Selenium Web Driver

I think everyone should be taught to code just like how we are taught to learn at least one language while we grow up. While a language such as English or Spanish enables an individual communicate with another, learning to code will empower an individual to better navigate this digital world powered by technology evolution. There are many compelling reasons for why this is important. Technology already plays a major role in all of our lives. And it’s reach is only going to increase and there’s no way around it. One might as well learn to code and make it work for us.

Any repeated task is ripe for automation. Repetition bores humans while Computers are immune to such boredom. AmazonBalanceReload is a simple project based in Scala that fires up a Web Browser, Logs the user in to Amazon, opens up the Balance Reload page on Amazon, fills in the balance to reload and submits it. This uses Selenium Web Driver which is a popular tool for writing Functional tests to a Web Based project. Let’s take a look at this simple implementation.

The main method in the AmazonAutoReload Scala object serves as the entry point to the application much like the main method in a Java class. This holds the necessary properties which alternatively be picked up from a property file or be passed as arguments when running the Object. But just to keep things simple, they are maintained inline in this implementation.

 def main(args: Array[String]): Unit = {

    val driver: WebDriver = new ChromeDriver()

    val n = 1 //Number of reloads
    val amountPerReload: String = "0.5" //In USD
    val email = "username" //Amazon email/phone number
    val password = "password" //Amazon password.


    val url = "https://www.amazon.com/ap/signin?_encoding=UTF8&ignoreAuthState=1&openid.assoc_handle=usflex&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.ns.pape=http%3A%2F%2Fspecs.openid.net%2Fextensions%2Fpape%2F1.0&openid.pape.max_auth_age=0&openid.return_to=https%3A%2F%2Fwww.amazon.com%2F%3Fref_%3Dnav_custrec_signin&switch_account="
    val email_field = "email"
    val password_field = "password"
    val submitBtn_field = "signInSubmit"

    login(driver, url, email_field, password_field, submitBtn_field, email, password)
    reload(driver, n, amountPerReload)

  }

 

The main method calls the login() method first while passing in the Web Driver instance among other things.

def login(driver: WebDriver, url: String, email_field: String, password_field: String, submitBtn_field: String, email: String, password: String): Int = {

    driver.get(url)

    val username_field_ele: WebElement = driver.findElement(By.name(email_field))
    username_field_ele.sendKeys(email)

    val password_field_ele = driver.findElement(By.name(password_field))
    password_field_ele.sendKeys(password)


    val submitBtn = driver.findElement(By.id(submitBtn_field))
    submitBtn.submit()
    return 0

  }

After the login is complete synchronously, reload() is then called which is just the front for calling the reloadOnce() based on the user input

def reload(driver: WebDriver, n: Int, amountPerReload: String): Int = {

    for (i <- 1 to n) {
      reloadOnce(driver, amountPerReload)
      Thread.sleep(10000)
    }

    return 0
  }

reloadOnce()

 def reloadOnce(driver: WebDriver, amountPerReload: String): Int = {

    driver.get("https://www.amazon.com/asv/reload/order?ref=asv_re_th_ftr_d")

    val manualReloadAmt = driver.findElement(By.id("asv-manual-reload-amount"))
    manualReloadAmt.sendKeys(amountPerReload)
    Thread.sleep(2000)
    manualReloadAmt.sendKeys((Keys.TAB))
    Thread.sleep(2000)

    val confirmPayment = driver.findElements(By.className("a-alert-content"))
    if (confirmPayment.size() > 0)
      confirmPaymentDetails(driver, "xxxx")


    Thread.sleep(2000)

    driver.findElement(By.id("form-submit-button")).submit()

    return 0
  }

Sometimes, Amazon triggers a payment verification step if it senses that something is fishy (Like trying to use this script to reload multiple times within a short period of time). The confirmPaymentDetails() takes care of that. This needs to further tuned to be production ready. But since this is for fun and for learning, this should suffice for now.

  def confirmPaymentDetails(driver: WebDriver, card: String): Int = {
    driver.findElement(By.id("asv-payment-edit-link")).click()
    Thread.sleep(2000)
    driver.findElement(By.id("pmts-id-2")).click()
    Thread.sleep(2000)
    driver.findElement(By.id("pmts-id-31")).sendKeys(card)
    Thread.sleep(2000)
    driver.findElement(By.id("pmts-id-33-announce")).submit()
    Thread.sleep(2000)
    driver.findElement(By.id("pmts-id-2")).click()
    Thread.sleep(2000)
    driver.findElement(By.id("asv-form-submit")).click()
    return 0

  }

Selenium is very powerful and it can be re purposed to automate practically anything we do in a browser (With some caveats!) such as checking your bank balances, paying all of your bills while logging in to each biller (Takes time to setup but it can be done) or any other repetitive task.

Source Code

https://github.com/batchu/AmazonBalanceReload.git

Java 8 and LocalDate serialization issues with Jackson mapper

Say you have a LocalDate attribute in one of your models in your Spring based RESTful Web Service and you need your clients to send and receive the LocalDate in some form of yyyyMMdd
LocalDate myDate;

Simply add the following dependency to your classpath (pom.xml)

   com.fasterxml.jackson.datatype
   jackson-datatype-jsr310
   2.8.6

And annotate your LocalDate attributes as

@JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyyMMdd")
LocalDate myDate;

But what if your project also uses older Date Api? (Say you are using both the original Date and new LocalDate)

Then you might have something like this to set the date format universally across the project to convert all Date instances

@Bean
 public MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter() {
     MappingJackson2HttpMessageConverter jsonConverter = new MappingJackson2HttpMessageConverter();
     ObjectMapper objectMapper = new ObjectMapper();
     DateFormat df = new SimpleDateFormat("yyyyMMdd");
     df.setTimeZone(TimeZone.getTimeZone("EST"));
     objectMapper.setDateFormat(df);
     jsonConverter.setObjectMapper(objectMapper);
     return jsonConverter;
 }

Jackson is still on JDK 6. So they have begun a modular plugin approach to support Java 8 features such as the new Date API (LocalDate) etc. To make Jackson play nice with both the old and the new APIs, simply update the above to

@Bean
 public MappingJackson2HttpMessageConverter mappingJackson2HttpMessageConverter() {
     MappingJackson2HttpMessageConverter jsonConverter = new MappingJackson2HttpMessageConverter();
     ObjectMapper objectMapper = new ObjectMapper()
           .registerModule(new ParameterNamesModule())
           .registerModule(new Jdk8Module())
           .registerModule(new JavaTimeModule());
     DateFormat df = new SimpleDateFormat("yyyyMMdd");
     df.setTimeZone(TimeZone.getTimeZone("EST"));
     objectMapper.setDateFormat(df);
     jsonConverter.setObjectMapper(objectMapper);
     return jsonConverter;
 }

Make sure that you also update your pom.xml with the necessary dependencies

   com.fasterxml.jackson.module
   jackson-module-parameter-names


   com.fasterxml.jackson.datatype
   jackson-datatype-jdk8


   com.fasterxml.jackson.datatype
   jackson-datatype-jsr310
   2.8.6

The first dependency in the list adds support for JDK8 datatypes before core Jackson can support them.

Documentation and Code samples of Open Source Projects should include Automated Tests

Let’s first take a quick detour:

The Software Engineering field thrives on Open Source Projects that is built for and by the people who are part of the community. This is a feature that is unique to the Software Engineering field and not quite common in other Creative or Science streams like Aeronautics or Biology etc. You do not see Open Source prototypes of Airplanes do you? May be because Software innately is Virtual and it makes it easy to duplicate and share. This ecosystem of constantly hitting a wall in some sphere of technology, coming up with revolutionary ideas to solve that problem and sharing that knowledge with the community is just beautiful. I’ve personally gained a lot of inspiration from projects like the Spring Framework, Angular etc. and myriad of cutting edge technologies that are constantly churned out by the Apache Software Foundation (ASF).

Back to the topic on hand:

Take a look at this Tutorial from Angular.io, the official home page of the popular Angular Framework. I’ve been closely following the evolution of the Angular Framework as well as how their documentation has also evolved in the past few years to its current state.

https://angular.io/docs/ts/latest/tutorial/

As you can see the above tutorial does a bang up job walking a reader through a small project to introduce core fundamentals of Angular. The tutorial is augmented by a Live Example. The live Example running on Plunkr shows the source code and its fantastic expect for the fact that there are no test cases bundled with it.

It’s the same deal with the Spring Framework and myriad of other Open Source projects.

https://spring.io/guides/

Here’s why live examples and code documentation must be bundled with test cases

  • To emphasize to the reader the value of writing automated tests
  • To “teach” the habit of writing test cases: Demonstrate the various ways the Code Sample can be tested. After all the goal of such source code is to help the reader borrow the ideas and implement them in their own projects. Showing well written test cases teaches the reader to write tests in their own implementations.
  • To verify your own work! The code sample itself needs to be tested. Do you really have to manually verify that your code works instead of using automated test cases to do the verification and validation?

I hope that Open Source projects seriously consider revising their approach towards their documentation and incorporate well written Automated tests in their Code Samples and wherever it makes sense. (Unit, Integration and others as appropriate)

The online HTML editor now supports the use of an external CSS file to be applied to format the content of the visual preview of your document.

Implementing custom methods for H2 Embedded Database

If you want to use an Embedded database such as H2 to unit test your code, you will

need to implement your method manually in H2 if it doesn’t supply one out of the box.

You can use Jooq to do this this

public class H2NewIDImpl {
    public static int counter = 0;
    public static int NEWID() throws SQLException {
        // Translate your T-SQL statements to jOOQ statements (if any)
       return counter++;
    }
}

and the sql to pick up the above class

CREATE ALIAS NEWID
FOR "net.prashu.H2NewIDImpl.NEWID";

Dealing with JPA Entities backed by SQL Queries that return no unique columns

Earlier I was working on a Spring Boot application where in I had to join multiple tables and map the returned columns to a JPA entity. But the problem was that none of the returned columns had unique values.

Here’s a snippet of how my JpaRepository looks like  (I put … to hide the parts of the query that are not relevant to this post)

@Repository
public interface UserJpaRepository extends JpaRepository {

    @Query(value="SELECT a.date_id as date_id...
           "FROM user a, address b, account c " +
            "WHERE ... a.id=?1",nativeQuery = true)
    List<User> getUserById(Long id);

}

But the way these three tables (user, address, account) are setup and the way the query is
structured, none of the columns returned had unique values in them. So I cannot annotate
any attribute in the User Entity with @id annotation. The getUserId(id) would always
return erroneous records. One approachto solve this is by adding newId() to be returned
as a column which is a randomly generated value for each returned row.

@Repository
public interface UserJpaRepository extends JpaRepository {

    @Query(value="SELECT
newId() AS col_id a.date_id as date_id...
            "FROM user a, address b, account c " +
            "WHERE ... a.id=?1",nativeQuery = true)
    List<User> getUserById(Long id);

}

So by adding colId to User and marking it as @id, the above method will return the results
properly.

Note that newId() method is available in MS SQL server. Every other standard RDMS provides
a way to do this.

Global Exception handling for Spring MVC

Proper exception handling in a REST service can help
  • Provide proper HTTP response codes and pertinent error messages to downstream clients
  • Logging custom exceptions can help with analytics (If logs are aggregated through a Log aggregation system such as Splunk, one can easily find out outcomes such as the number of times a specific error has occurred etc.)
Here’s one of many ways to raise custom exceptions and handle them globally using Spring @ControllerAsdvise
Here’s a REST URL mapping that retrieves a User resource by id in this controller
 
 @GetMapping(“/user/{id}”)
    public ResponseEntity<?> getUser(@PathVariable Long id) throws NoMatchingUserException {
        return ResponseEntity.ok(userService.findById(id));
    }
The method throws a custom NoMatchingUserException which can be found here
@ResponseStatus(value = HttpStatus.NOT_FOUND, reason = “No such User”)
public class NoMatchingUserException extends UserGenericException {
    public NoMatchingUserException(String message) {
        super(message);
        this.setHttpStatus(HttpStatus.NOT_FOUND);
    }
}
Now let’s look at the Servicemethod that is called by the Controller
    @Override
    public User findById(Long id) throws NoMatchingUserException {
        User user = repository.findById(id);
        if (user == null)
            throw new NoMatchingUserException(“No user found with id “ + id);
        return user;
    }
Line 37 throws the custom NoMatchingUserException with a customized error message.
The above configuration cleanly returns a useful error message to the client during such error condition
Here’s a snapshot of a successful response (When calling http://localhost:8080/user/1)
{id“: 1,firstName“: Jack,lastName“: Bauer}
Here’s a snapshot of an erroneous request (user ID 6 is not found in the system) and its corresponding response (When calling http://localhost:8080/user/6)
{ERROR“: No user found for id 6}
 
If you look in to the network tab of your browser console (Chrome DEV tools -> Network) you can also see that the response error code is properly set to 404.
This recipe can be expanded and generalized further. May be the custom exceptions hold the HTTP status code pertinent to the error condition and bubble it up for the Exceptionhandler to retrieve it and set the response error code. In such scenario, we can have one global generic Exceptionhandler which simplifies the solution further.
The source code for the above can be found here. Simply clone the repository and run the Application.java from your IDE to bootstrap the application.