Aiandreev.Data Lab.Hard Skills

Meer zoals dit

Agile Infrastructure

door Paweł Badeński

Nike From Supply Chain Crisis to Sustainability Leadership

door America Harris

Factors that Influence Liveability

door Emma Bozzo

Organizing and Representing Data: Visual Representations

door Katie Morrison

Aiandreev.Data Lab.Hard Skills

Agile & Communication

Presentations

LaTeX Beamer

Google Presentations

MS PowerPoint [J]

Diagramms

Figma

Mindomo [J]

PlantUML

Miro [J]

MS Visio

Gliffy

Draw.io [J]

Corporate Social Network

Yammer [J]

Communication

MS Outlook [J]

MS Teams [M]

Knowlege sharing

Confluence [M]

Task trackers

Trello

Jira [J]

IT Infrastructure

IaC

AWS CloudFormation

Terraform

Ansible [J]

Mail

AWS Workmail

Yandex Mail

Google Gmail

Monitoring

Prometheus

Kibana

Grafana

Zabbix

DNS

WAN

SSL Certificates

Multi domain

Wildcard

OV

EV

DV

DNS Records

PTR

NX

MX

CNAME

AAA

A

Zones

Domain

Structure

Routing delegation

Registration

Server

Yandex DNS

Google Public DNS

AWS Route 53

LAN

Load Balancer

HAProxy

Citrix ADC

AWS Elastic Load Balancing

Nginx

Azure Traffic Manager

Authorization

AWS IAM

LDAP [J]

MS Active Directory

Hosting & Serverless calculation

Windows [J]

Linux [J]

Containers

Container hosting

Kubernetes

Azure Databricks

AWS ECS

Docker

docker-compose

Windows

Selectel

SberCloud

Ya.Cloud

AWS Lightsail

AWS EC2

Azure Windows Server

Linux

Data Science

BI

Fine BI

Visiology

Apache Superset

Ya Datalens

Metabase

reDash

Spotfire

Tableau

Streamlit

Qlik View [M]

Qlik Sense [M]

MS Power BI

Data Mining

MS Excel [M]

SQL [M]

Plotly [J]

MathPlotLib [J]

Seaborn [J]

polars

Pandas [J]

NumPy [J]

numba

Machine Learning

Reinforcement learning

Temporal difference

Dynamic programming

Monte Carlo

Bandit

Recommendation Systems

Collaborative Filtering

Computer Vision

PyTorchCV

timm

Pytesseract

SimpleITK

Mahotas

Detection / Segmentation

pytorch-toolbelt

segmentation-models

detectron2

Image manipulation

Scikit-Image

Pillow

OpenCV

NLP

Emdeddings

fasttext

Gensim

Quadrant

Faiss

HuggingFace

pyMorphy2

spaCy

Textblob

Razdel

Natasha

NLTK

Deep learning

Jax

Keras

PyTorch

PyTorch Lightning

TensorFlow

Classic ML

Gradient Boosting

XGBoost

CatBoost

LightGBM

Vowpal Wabbit

Scikit-Learn

Linear models

Lasso

Ridge

Logistic Regression

Clusterization

Agglomerative

DBSCAN

KMeans

EDA

sweetviz

pandas-profiling

Statistics

Statsmodel

Pingouin

SciPy

Software Engineering

Python [J]

Deploy & Code Maintanance

MLOps

Serving

FastAPI

https://fastapi.tiangolo.com/tutorial/first-steps/

flask

bentoml

Experiment tracking

MLFlow

https://www.mlflow.org/docs/latest/quickstart.html

ClearML

https://clear.ml/docs/latest/docs/getting_started/ds/ds_first_steps

Data tracking & Quality

pydantic

pandera

CML

DVC

https://dvc.org/doc/start

Code quality

pre-commit

Formatters

black

Testing

unittest

hypothesis

pytest

pytest-coverage

Linters

pycodestype

mypy

Flake8

wemake

Git

GitLab [J]

Bitbucket [J]

GitHub [J]

CI / CD

Team City

Jenkins

Github actions

https://docs.github.com/en/actions/learn-github-actions

Gitlab CI CD

https://docs.gitlab.com/ee/ci/

Algorithms & Data Structures [J]

Data Engineering

MQ

IBM MQ

AWS MQ

AWS SQS

Kafka

Rabbit

ETL

AWS DMS

AWS Glue

Informatica

Databricks

MS SSIS

MS Data Factory

TalenD

Pentaho DI [J]

Languages

Go

R

https://stepik.org/course/497/syllabus

Java

Так себе курс, подойдет только, если вы совсем ничего не знаете о java

https://stepik.org/course/497/syllabus

Scala

Catz

Python

connectorX

ScraPy

dask

https://docs.dask.org/en/stable/10-minutes-to-dask.html

lxml

BeautifulSoup

bonobo

psycopg2 / 3

pandas [J]

sqlalchemy [J]

requests

Data Storage

Storage

Protocols

HTTP

WebDAV

S3

SCP

Использование на уровне юзера

Копирование с локальной тачки на ремоут тачку

scp local_path username@host:remote_path

Копирование с ремоут тачки на локальную

scp username@host:remote_path local_path

SFTP

FTP

Sber Disk

Ya Disk

MS OneDrive

MS Sharepoint

Minio S3

Установка minio

https://docs.min.io/docs/minio-docker-quickstart-guide.html

MS Blob Storage

Google Drive

AWS S3

NON - RDBMS

Arango DB

Neo4J

MariaDB

Google Firebase

AWS Dynamo DB

Apache Cassandra

Apache Ignite

MS Cosmos DB

MongoDB

Apache Hadoop

Для пользователя на самом деле достаточно знать 2 команды

hdfs -get remote local

hdfs -put local remote

Архитектура hdfs (Java Api изучать не надо)

https://stepik.org/lesson/15482/step/1?unit=4233

RDBMS

Distributed

On premise

Clickhouse

Apache Hive

Citus

ArenaData

YDB

Snowflake

Google BigQuery

MS Synapse

AWS Redshift

Classic

https://www.sql-ex.ru/

Oracle

MySQL [J]

PostgreSQL [M]

MS SQL Server [M]

Курс по Базам Данных, которые читали в УрФУ 2022

https://www.youtube.com/playlist?list=PLuYsCpx95Allwadi6NMeUYjGg31g7UsPP

Pipeline Orchestration

Cloud

Azure Data Factory

Google Cloud Composer

AWS Step Functions

Tools

Metaflow

Dagster

https://docs.dagster.io/guides

Rundeck

Prefect

Luigi

https://luigi.readthedocs.io/en/stable/

Airflow [J]

https://airflow.apache.org/docs/apache-airflow/stable/howto/index.html

Astronomer - очень полезный ресурс!

https://www.astronomer.io/guides/

Cron

https://ostechnix.com/a-beginners-guide-to-cron-jobs/