Web services using Python 

 python shell:

Prerequisites:

Basic working knowledge of MongoDB and Apache Spark.  Refer to the MongoDB documentation and Spark documentation for more details.

Running MongoDB instance (version 3.4.7 or later).

Spark 2.2(upgraded version)

Getting Started:

Python Spark Shell:When starting the pyspark shell, you can specify,

the --packages option to download the MongoDB Spark Connector package. The following

package is available:

mongo-spark-connector_2.11 for use with Scala 2.11.x

the --conf option to configure the MongoDB Spark Connnector. These settings configure

the SparkConf object.

The following example starts the pyspark shell from the command line:

./bin/pyspark --conf "spark.mongodb.input.uri=mongodb://127.0.0.1/test.myCollection?readPreference=primaryPreferred" \

              --conf "spark.mongodb.output.uri=mongodb://127.0.0.1/test.myCollection" \

              --packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.0

eg:  bin/pyspark --conf "spark.mongodb.input.uri=mongodb://127.0.0.1/dhcmain.doctor?readPreference=primaryPreferred" \

--conf "spark.mongodb.output.uri=mongodb://127.0.0.1/test.myCollection" \

--packages org.mongodb.spark:mongo-spark-connector_2.11:2.2.0

Note: Go to spark home i.e,

 cd spark_file
 paste the above command, which will open as 'pyspark shell- >>>' automatically.

* The spark.mongodb.input.uri specifies the MongoDB server address (127.0.0.1), the database to connect (test),

and the collection (myCollection) from which to read data, and the read preference.

* The spark.mongodb.output.uri specifies the MongoDB server address (127.0.0.1), the database to connect (test), and the collection (myCollection) to which to write data. Connects to port 27017 by default.

* The packages option specifies the Spark Connector’s Maven coordinates, in the format groupId:artifactId:version.

The examples here will use this database and collection.

Create a SparkSession Object:

If you specified the spark.mongodb.input.uri and spark.mongodb.output.uri configuration options when you started pyspark,

the default SparkSession object uses them. If you’d rather create your own SparkSession object from within pyspark,

you can use SparkSession.builder and specify different configuration options.

>>>from pyspark.sql import SparkSession

>>>my_spark = SparkSession \

    .builder \

    .appName("myApp") \

    .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.coll") \

    .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll") \

    .getOrCreate()

eg:

>>>spark = SparkSession \

    .builder \

    .appName("demography mapper") \

    .getOrCreate()

>>>df_user = spark.read.format("com.mongodb.spark.sql.DefaultSource")\

    .option("spark.mongodb.input.uri", "mongodb://localhost:27017/dhcmain.doctor").load()

*You can use a SparkSession object to write data to MongoDB, read data from MongoDB, create DataFrames, and perform SQL

operations.

such as,

>>>print(df_user.printSchema())

Read from MongoDB:

You can create a Spark DataFrame to hold data from the MongoDB collection specified in the spark.mongodb.input.uri option which your SparkSession option is using.

Consider a collection that contains documents:

Assign the collection to a DataFrame with spark.read() from within the pyspark shell.

>>>df = spark.read.format("com.mongodb.spark.sql.DefaultSource").load()

Before you can run SQL queries against your DataFrame, you need to register a temporary table.

The following example registers a temporary table called temp, then uses SQL to query to fetch data from the  temporary table created:

>>>df.registerTempTable(“temp”)

>>>result = spark.sql(“select * from temp”)

>>>res.show()

That’s all in python shell, Now try different  use cases in similar way.

Good Luck☺